create account

RE: Particle physics @ utopian-io - Objects isolation, histogramming and a first task request by lemouth

View this thread on: hive.blogpeakd.comecency.com

Viewing a response to: @effofex/re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180618t200008852z

· @lemouth ·
$0.08
Here are answers to your questions. Do not hesitate to come back to me if necessary.

> Are the histogram names within the *.saf file unique (will "ptj1" only show up once?)

All histograms are concatenated within a given SAF file.

>There's no info on the x, y, and title in the file, what are the defaults you'd like?

The title is given between quotes, right after the description. The axis names are not passed. You can leave them blank. However, it may be nice to allow the user to give them.

> Are the under/overflow bins shown in the histogram?

It would be nice to leave this option to the user.

> Are you looking for a python module which can be called from code or a command line tool?

Both can be done quite easily.

> Do you want this to produce an image file or mataplotlib object which can be further edited?

Once again, both. The matplotlib file is important to allow the user to tune it to the layout he/she wants.

> The value of bins 1, 2, and 3 are 1, 3x10-4, and 5x10-4, respectively. Further, we don't need to show this split in the histogram. Correct?

The total number to put in a bin corresponds to the first column minus the second one.
👍  ,
properties (23)
authorlemouth
permlinkre-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180619t132533827z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-19 13:25:36
last_update2018-06-19 13:25:36
depth3
children18
last_payout2018-06-26 13:25:36
cashout_time1969-12-31 23:59:59
total_payout_value0.060 HBD
curator_payout_value0.017 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length1,165
author_reputation338,011,164,701,274
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id61,359,283
net_rshares37,253,181,047
author_curate_reward""
vote details (2)
@effofex ·
Thanks for the clarifications. I think I understand it enough to get an implementation coded up.
properties (22)
authoreffofex
permlinkre-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180621t021158643z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-21 02:12:00
last_update2018-06-21 02:12:00
depth4
children1
last_payout2018-06-28 02:12:00
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length96
author_reputation14,429,105,750,792
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id61,573,274
net_rshares0
@lemouth ·
I will double check the code anyways :)
properties (22)
authorlemouth
permlinkre-effofex-re-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180621t184922495z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-21 18:49:24
last_update2018-06-21 18:49:24
depth5
children0
last_payout2018-06-28 18:49:24
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length39
author_reputation338,011,164,701,274
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id61,678,022
net_rshares0
@effofex ·
I've got some good news.  This is still janky as heck, but I've got a python tool which parses the saf file, converts it into a pandas dataframe, and allows me to spit out some histograms.  

The janky part is that the histograms are still essentially a proof-of-pipeline, matplotlib has turned out to be very different (not bad, just different) from the plotting language (ggplot2) I am used to.

![first_histo.png](https://cdn.steemitimages.com/DQmVtg8unfSysiAfz7mJLYpq2p5A41KbsgLKcJVPM4xApkf/first_histo.png)

This image will get better between now and Friday. My plan to is push my local repo to a fork of ma5 on my github account, then send you a pull request on Friday. Based on how that goes and your feedback, then write up a post.  

The downside is that I probably won't have time for 1c, but I can deal with that.  It's been very informative using pandas and matplotlib on non-toy data.
properties (22)
authoreffofex
permlinkre-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180628t021631552z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"image":["https://cdn.steemitimages.com/DQmVtg8unfSysiAfz7mJLYpq2p5A41KbsgLKcJVPM4xApkf/first_histo.png"],"app":"steemit/0.1"}
created2018-06-28 02:16:30
last_update2018-06-28 02:16:30
depth4
children11
last_payout2018-07-05 02:16:30
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length897
author_reputation14,429,105,750,792
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,555,997
net_rshares0
@lemouth ·
Super! This looks very nice. Just two comments:
 - Would it be possible to have different figures for each histogram?
 - What do you mean by a fork to MA5? Can't your code be used externally?

I can give one more week for 1c, as I still haven't found the time to start writing 1d  ^^
properties (22)
authorlemouth
permlinkre-effofex-re-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180628t054840334z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-28 05:48:48
last_update2018-06-28 05:48:48
depth5
children10
last_payout2018-07-05 05:48:48
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length283
author_reputation338,011,164,701,274
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,573,271
net_rshares0
@effofex · (edited)
I can certainly make it multiple figures, i was actually pondering that over morning coffee.

The code exists as a standalone thing, but utopian would like your github ma5 to be the main repo, so I am developing it in a new directory under tools. This also helps with the criterion that a task is accepted by an author by incorporating a pull request.
properties (22)
authoreffofex
permlinkre-lemouth-re-effofex-re-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180628t131821998z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-28 13:18:42
last_update2018-06-28 13:19:45
depth6
children4
last_payout2018-07-05 13:18:42
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length351
author_reputation14,429,105,750,792
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,615,968
net_rshares0
@effofex ·
Pull request is live!

Easiest way to get a feel for it is , from the HistoGen directory:
`python src\HistoGen.py test\data\example_histos.saf --t`

You can get help with:
`python src\HistoGen.py -h` (or by asking me ;) )

```
usage: HistoGen [-h] [-o OUTDIR] [--verbose] [-t] [-v] infile

Generate some MadAnalysis5 histograms from an associated SAF file.

positional arguments:
  infile                Path to a a valid SAF file describing histogram data

optional arguments:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        Location in which to store generated histograms
                        ((default is current dir)
  --verbose             Provide lots of output during run (default is silent)
  -t, --terse           Provide some output during run (default is silent)
  -v, --version         show program's version number and exit
```

looking at `HistoGen.py` should also give you a feel for how this can be used a code module instead of as a CLI tool. 

Should get output as below:
![Fig_1.png](https://cdn.steemitimages.com/DQmdRDa4y4gzKJHuGQs8dvaz5GNmYY9tKRFCbtQV2KoJWa3/Fig_1.png)
![Fig_2.png](https://cdn.steemitimages.com/DQmV7mRrf5XAzSv1xfDy7xv7Ya5L2vNgY7AsyPWtgRtNLMq/Fig_2.png)
![Fig_3.png](https://cdn.steemitimages.com/DQmUd91Lc6aR9Kh17cqQjXYZBDK46GUvJjdDiEcbeosiFCW/Fig_3.png)
![Fig_4.png](https://cdn.steemitimages.com/DQmRmK6UCiQXcENs7YwCfVxEZuJVpFXWW4YcmEfFgL6vWtN/Fig_4.png)
properties (22)
authoreffofex
permlinkre-lemouth-re-effofex-re-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180629t034551633z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"image":["https://cdn.steemitimages.com/DQmdRDa4y4gzKJHuGQs8dvaz5GNmYY9tKRFCbtQV2KoJWa3/Fig_1.png","https://cdn.steemitimages.com/DQmV7mRrf5XAzSv1xfDy7xv7Ya5L2vNgY7AsyPWtgRtNLMq/Fig_2.png","https://cdn.steemitimages.com/DQmUd91Lc6aR9Kh17cqQjXYZBDK46GUvJjdDiEcbeosiFCW/Fig_3.png","https://cdn.steemitimages.com/DQmRmK6UCiQXcENs7YwCfVxEZuJVpFXWW4YcmEfFgL6vWtN/Fig_4.png"],"app":"steemit/0.1"}
created2018-06-29 03:45:48
last_update2018-06-29 03:45:48
depth6
children4
last_payout2018-07-06 03:45:48
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length1,461
author_reputation14,429,105,750,792
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,691,379
net_rshares0
@irelandscape ·
Hi, sorry if this is strange question but I don't understand the resulting bin data values in the SAF file.

I thought that in the histogram the bin value should be whole numbers representing the number of particles matching a particular value range.

Am I wrong about this?
properties (22)
authorirelandscape
permlinkre-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180622t155802354z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-22 15:58:03
last_update2018-06-22 15:58:03
depth4
children3
last_payout2018-06-29 15:58:03
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length274
author_reputation15,380,678,988,494
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id61,790,525
net_rshares0
@lemouth ·
Apologies for a very late answer... Complicated times for me at the moment...

> I thought that in the histogram the bin value should be whole numbers representing the number of particles matching a particular value range.

It depends what you want to plot. 

If you are interested in the transverse momentum distribution of all jets, then each jet of the event will contribute to the histogram (to potentially different bins). On the other hand, you may be interested in the transverse momentum distribution of the leading (or first jet) of the events. Then, only the first jet matters and will yield an entry in the histogram. 

We can also be interested in plotting global variables, like the total activity in the events, which do not depend on the actual number of particle but instead of the sum of their energy.

Does it clarify?
properties (22)
authorlemouth
permlinkre-irelandscape-re-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180627t053153543z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-27 05:32:00
last_update2018-06-27 05:32:00
depth5
children2
last_payout2018-07-04 05:32:00
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length836
author_reputation338,011,164,701,274
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,439,048
net_rshares0
@irelandscape ·
Thanks, yes I think that clarifies it.
I will submit corrections shortly, probably today.
Cheers!
properties (22)
authorirelandscape
permlinkre-lemouth-re-irelandscape-re-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180627t063939661z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-27 06:39:39
last_update2018-06-27 06:39:39
depth6
children1
last_payout2018-07-04 06:39:39
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length97
author_reputation15,380,678,988,494
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,445,452
net_rshares0