Present: Lee, Tom (scribe), Nic, Mike, Tim (hangouts)
- Nic: Xover bias
- Tom: Benchmarks
- Bill: EHC
- Talk about who might be attending
- What should we write up?
- Maybe simplification as post-run generalization improver?
- Maybe in trees?
- Maybe non-hill climbing using silencing?
- Involuntary transactins
Present: Lee, Nic, Lance, Tom (scribe), Ben, Beryl, Tim, Wren, Eddie
Finishing some Student Project Ideas from Last Week
Eddie – He’s tried some stuff with Brevis and Quill. He might next work some with Pucks.
Ben – Not sure yet. Maybe something A-Life-y?
Tom’s Benchmark Problems
Should we include problems beyond iJava and Yuriy’s paper?
- We should have a citable source outside our work for the problem — i.e. not “we made it up”. Otherwise, probably “yes”.
- FizzBuzz (kata) – yes
- Bowling (kata) – maybe
- Factorial – maybe (too easy?)
- WC – yes
- US Change – maybe (need citable source)
- Tom will consult rest of committee.
Wren is going to try some things and consult others to try to see if a solution already exists.
Homology of Population
- Levenshtein distance of pairs of programs seems like good measurement.
- Likely can’t test all pairs, since this will usually be about 1 million Levenshtein distance computations. This will probably be too slow.
- Instead, sample random pairs of individuals — maybe on the order of 1000 to 10000.
- May need to do validation to make sure the sample results really reflect the population.
- Report: median, quartiles, mean, standard deviation.
Nic McPhee Presents: Analysis of Ancestry in GP with a Graph DB
- Using Neo4j – a graph database – to store ancestry info throughout a GP run.
- Makes ancestry info easy to collect and query.
- NeoCons – Clojure with Neo4j
- Cool stats can be queried from the database!
- Could be neat to look at common ancestries using different parent selection methods!
Present: Lee, Tom, Nic, Lance, Mike, Tim, Bill, Kwaku (virtual), Karthik (virtual)
He’d gladly have help coding and doing experiments on program synthesis benchmark problems.
Clojush 2 is now live. It adds a linear representation for genomes, which is translated into regular Push programs for execution. People should try it out and report bugs!
Documentation of Recent Push
Mike (and Nic) will be working on documenting the use of Push as they get to know it.
Bladder Cancer Data Set
Clojush 2 has the training error down near 70. Lee hasn’t figured out how to control overfitting yet.
Discussion of Paper: Evolving Neural Networks That Are Both Modular and Regular: HyperNeat Plus the Connection Cost Technique Categories and Subject Descriptors
- There are different kinds of modularity, and what we want is different from what they present.
- Evolving an interface for functions is hard!
- Push environments might be helpful here.
- Varying the “environment” of the problem might help.
- Lexicase is sort of like varying the environment. Maybe?
- It seems like it would be reasonable to add a test case that tries to improve modularity based on the compressibility of the program.
- We’d likely be interested in minimizing the amount a program is compressed. Something like: (compressed size of program)/(uncompressed size of program)
Present: Lee, Tom (scribe), Tim
NO MEETING NEXT WEEK!
For the following week: Read the paper Evolving Neural Networks That Are Both Modular and Regular: HyperNeat Plus the Connection Cost Technique Categories and Subject Descriptors, which will be sent to the email list by Tom.
- Lee touted lexicase as potentially good for their work
- We got a bladder cancer data set to try things with.
- Lee did some initial hackings, starting with simple trees, and is now trying Clojush
- Tim will be working on this.
WC + Tags
No difference between tags and no tags 🙁
It could be that instructions are no-oping near the beginning of programs, either because of a bug or just because the required stacks are empty. We could track the number of no-oping instructions to see if it increases throughout a run. It could be interesting to see what percent of instructions noop at each program position during each generation — you could have generations on the x-axis and program positions on the y-axis, and a heatmap showing what percent of instructions noop.
- Present: Lee, Tom (scribe), Tim, Karthik (remote), Bill (remote)
- Tom will practice GECCO talk next week
- Frode should send us GECCO slides shortly
Plush genomes -> Push programs
- Do we want any Push program to be able to be represented by a Plush genome?
- To do so, we have new instructions:
- meta_open – opens new paren
- meta_splice_backward – deleted previous paren pair
Hill-climbing vs. Genetic Operators
- e.g. simplification, epigenetics, etc.
- Is it useful to test and revert (using a fitness evaluation), or is just making the change and letting selection work it out just as good?
- Maybe Bill or Tom should experiment with this — EHC vs. using epigenetic genetic operators with more generations to make up for EHC’s extra evaluations.
- Simplest is to never do hill climbing, just have genetic operators.
- Decision: Implement a wrapper argument (like :make-reverting) that can be used in genetic operator pipelines that cases the next operator to be reverted if not better than its input program.
Issue of what to do with child programs that exceed size limits
Decision: implement argument with these options:
- return clone of one parent (default, as previously)
- return empty program
- return random program
- return one parent with some part(s) deleted
Present: Lee, Tom (scribe) Karthik, Bill, Omri
We should invite Ryan Shams to a lab meeting.
Lee’s A-Life-y thing
- Lazy can be annoying
- footnote – agent handling of errors in Clojush should get better
- Reducers might be useful
Bill’s Wind Turbine Results
- Lots of data
- Some promising results so far
- May try co-evolutionary methods
Closeline Linear Push
So far, it is unclear whether this is “The Future” of Push, or just another tool that could be selectively applied depending on the problem. If selective, it would be nice to have a switch in Clojush to turn it on and off. If not, then we should make a final version so others can use it.
Either way, eventually the code should be cleaned up. In particular, maybe all epigenetic lines should be stored in a map of them, so that we can have closelines, offlines, etc.
Less Than 2000 Problem
- There might be something interesting to look into in comparing the complexity of the data to the complexity of the evolved programs. If programs are more complex, then maybe they don’t generalize?
- General principle of “start small” might help somewhere here:
- Small population
- small number of test cases
- small program sizes
- Offline -> epigenetics might be helpful for getting smaller programs of the right size.
Present: Lee, Tom (scribe), Frode, Karthik (virtual)
Notes and Lab Page
- Get Karthik up to speed for posting lab notes
- Update lab pages when possible: especially fly section
Simplification Paper – Journal Article
- Put together poster paper and Frode’s workshop paper
Laziness Can Get You Into Trouble
Lee’s Heisenbug memory leak
Tom’s Selection Comparisons Lexicase/Tourney
- Try graphing lexicase selections with total error on the x-axis instead of rank
- Maybe look at minimum error by test case vectors to see if they are similar or different between runs. Could (but not necessarily) show whether or not the runs are exploring similar or different search spaces.
Present: Lee, Tom (scribe), Bill, Charles Ofria
Note: Student travel grants for GECCO are now open for application.
- Charles suggests using (average) Hamming distance of output vectors instead of behavioral diversity to measure the population diversity
- This will do a better job of measuring the coverage of the phenotypic space (or “trait coverage”)
- could just sample this, don’t have to compare every pair of individuals
Discussion of co-solvability paper
- Very related in motivation to lexicase
- lexicase is mostly different in that it can select individuals that are really bad at a few test cases as long as they are really good at others.
- Also, co-solvability requires binary test cases, without an obvious way to use on non-binary cases
Attending: Lee, Tom (scribe), Bill, Omri, Kartik (virtual), Kwaku (virtual)
Karthik – can probably connect Clojush and Z3 through Java API.
Lexicase Paper Rejection
- Some possible additions ideas:
- Investigate how much of pop is let around after each step of lexicase
- time runs for this paper
- Lexicase speedup idea: use a lexicase comparator, and then find optimum in population
- To do this, would need a lexicase comparator that takes 2 individuals (or error vectors) as well as a test case ordering, and returns the individual with better initial cases based on the ordering
- Then, you can use the comparator to find the “max” (or “min”) of the population – basically, use it with reduce to find best individual for that ordering
- We should include this as a possibility in paper, but probably not implement it.
- Thoughts on niching/fitness sharing:
- We should definitely enhance our discussion with related techniques, but not necessarily implement and compare with them
- We should clarify how lexicase is different
- If we find one option that is common and simple, we could try it out if we have time.
Attending: Lee, Omri, Frode, Tom (hangout, scribe), Kwaku (hangout), Karthik (hangout)
- We should think about what to do for GPTP.
- No concrete plans for Moshe Sipper.
Omri Div III
Will involve QGame and using it for educational purposes.
May or may not use WC data depending on space constraints.
- Last gasp for trying to get Calc to work
- Tried, didn’t work:
- Simplified instruction set
- “behaviors” = “domains” test case generation
- Put “old” test cases at front of lexicase order
- New things to try:
- Two alignment deviations: most of the time use 0, but sometimes use large (like 100)
- ULTRA tags by running one parent program, and then when run second, when it tags something instead ULTRA what it would tag with whatever is already tagged there.
- Similarly, could take each parent and create its tagspace, and then re-create a program from the tagspace, lining up positions of tagging instructions for the same tag by padding with ‘ultra-padding, which will be removed after ULTRA.
- Maybe try trivial geography with lexicase.