create account

RE: Particle physics @ utopian-io - Objects isolation, histogramming and a first task request by effofex

View this thread on: hive.blogpeakd.comecency.com

Viewing a response to: @lemouth/re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180619t132533827z

· @effofex ·
I've got some good news.  This is still janky as heck, but I've got a python tool which parses the saf file, converts it into a pandas dataframe, and allows me to spit out some histograms.  

The janky part is that the histograms are still essentially a proof-of-pipeline, matplotlib has turned out to be very different (not bad, just different) from the plotting language (ggplot2) I am used to.

![first_histo.png](https://cdn.steemitimages.com/DQmVtg8unfSysiAfz7mJLYpq2p5A41KbsgLKcJVPM4xApkf/first_histo.png)

This image will get better between now and Friday. My plan to is push my local repo to a fork of ma5 on my github account, then send you a pull request on Friday. Based on how that goes and your feedback, then write up a post.  

The downside is that I probably won't have time for 1c, but I can deal with that.  It's been very informative using pandas and matplotlib on non-toy data.
properties (22)
authoreffofex
permlinkre-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180628t021631552z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"image":["https://cdn.steemitimages.com/DQmVtg8unfSysiAfz7mJLYpq2p5A41KbsgLKcJVPM4xApkf/first_histo.png"],"app":"steemit/0.1"}
created2018-06-28 02:16:30
last_update2018-06-28 02:16:30
depth4
children11
last_payout2018-07-05 02:16:30
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length897
author_reputation14,429,105,750,792
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,555,997
net_rshares0
@lemouth ·
Super! This looks very nice. Just two comments:
 - Would it be possible to have different figures for each histogram?
 - What do you mean by a fork to MA5? Can't your code be used externally?

I can give one more week for 1c, as I still haven't found the time to start writing 1d  ^^
properties (22)
authorlemouth
permlinkre-effofex-re-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180628t054840334z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-28 05:48:48
last_update2018-06-28 05:48:48
depth5
children10
last_payout2018-07-05 05:48:48
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length283
author_reputation338,011,164,701,274
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,573,271
net_rshares0
@effofex · (edited)
I can certainly make it multiple figures, i was actually pondering that over morning coffee.

The code exists as a standalone thing, but utopian would like your github ma5 to be the main repo, so I am developing it in a new directory under tools. This also helps with the criterion that a task is accepted by an author by incorporating a pull request.
properties (22)
authoreffofex
permlinkre-lemouth-re-effofex-re-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180628t131821998z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-28 13:18:42
last_update2018-06-28 13:19:45
depth6
children4
last_payout2018-07-05 13:18:42
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length351
author_reputation14,429,105,750,792
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,615,968
net_rshares0
@lemouth · (edited)
I am not sure to understand the utopian request. This addon was supposed to stay independent of madanalysis, as it is only useful once the working directory has been created. I will see with utopian.
properties (22)
authorlemouth
permlinkre-effofex-re-lemouth-re-effofex-re-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180629t161401914z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-29 16:14:06
last_update2018-06-29 16:23:30
depth7
children3
last_payout2018-07-06 16:14:06
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length199
author_reputation338,011,164,701,274
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,761,314
net_rshares0
@effofex ·
Pull request is live!

Easiest way to get a feel for it is , from the HistoGen directory:
`python src\HistoGen.py test\data\example_histos.saf --t`

You can get help with:
`python src\HistoGen.py -h` (or by asking me ;) )

```
usage: HistoGen [-h] [-o OUTDIR] [--verbose] [-t] [-v] infile

Generate some MadAnalysis5 histograms from an associated SAF file.

positional arguments:
  infile                Path to a a valid SAF file describing histogram data

optional arguments:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        Location in which to store generated histograms
                        ((default is current dir)
  --verbose             Provide lots of output during run (default is silent)
  -t, --terse           Provide some output during run (default is silent)
  -v, --version         show program's version number and exit
```

looking at `HistoGen.py` should also give you a feel for how this can be used a code module instead of as a CLI tool. 

Should get output as below:
![Fig_1.png](https://cdn.steemitimages.com/DQmdRDa4y4gzKJHuGQs8dvaz5GNmYY9tKRFCbtQV2KoJWa3/Fig_1.png)
![Fig_2.png](https://cdn.steemitimages.com/DQmV7mRrf5XAzSv1xfDy7xv7Ya5L2vNgY7AsyPWtgRtNLMq/Fig_2.png)
![Fig_3.png](https://cdn.steemitimages.com/DQmUd91Lc6aR9Kh17cqQjXYZBDK46GUvJjdDiEcbeosiFCW/Fig_3.png)
![Fig_4.png](https://cdn.steemitimages.com/DQmRmK6UCiQXcENs7YwCfVxEZuJVpFXWW4YcmEfFgL6vWtN/Fig_4.png)
properties (22)
authoreffofex
permlinkre-lemouth-re-effofex-re-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180629t034551633z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"image":["https://cdn.steemitimages.com/DQmdRDa4y4gzKJHuGQs8dvaz5GNmYY9tKRFCbtQV2KoJWa3/Fig_1.png","https://cdn.steemitimages.com/DQmV7mRrf5XAzSv1xfDy7xv7Ya5L2vNgY7AsyPWtgRtNLMq/Fig_2.png","https://cdn.steemitimages.com/DQmUd91Lc6aR9Kh17cqQjXYZBDK46GUvJjdDiEcbeosiFCW/Fig_3.png","https://cdn.steemitimages.com/DQmRmK6UCiQXcENs7YwCfVxEZuJVpFXWW4YcmEfFgL6vWtN/Fig_4.png"],"app":"steemit/0.1"}
created2018-06-29 03:45:48
last_update2018-06-29 03:45:48
depth6
children4
last_payout2018-07-06 03:45:48
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length1,461
author_reputation14,429,105,750,792
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,691,379
net_rshares0
@effofex · (edited)
Written in a Python 3 environment (3.5), external dependencies include:
* matplotlib
* pandas
* numpy
* seaborn
* pytest

running pytest from the `HistoGen` dir should work. Right now the unit tests simply check to see if I'm reading the SAF file correctly.
properties (22)
authoreffofex
permlinkre-effofex-re-lemouth-re-effofex-re-lemouth-re-effofex-re-effofex-re-lemouth-particle-physics-utopian-io-objects-isolation-histogramming-and-a-first-task-request-20180629t034756105z
categoryutopian-io
json_metadata{"tags":["utopian-io"],"app":"steemit/0.1"}
created2018-06-29 03:47:51
last_update2018-06-29 03:53:33
depth7
children3
last_payout2018-07-06 03:47:51
cashout_time1969-12-31 23:59:59
total_payout_value0.000 HBD
curator_payout_value0.000 HBD
pending_payout_value0.000 HBD
promoted0.000 HBD
body_length257
author_reputation14,429,105,750,792
root_title"Particle physics @ utopian-io - Objects isolation, histogramming and a first task request"
beneficiaries[]
max_accepted_payout1,000,000.000 HBD
percent_hbd10,000
post_id62,691,547
net_rshares0