- Closures work like they do elsewhere. The environment in which it was
defined, not executed, determines the value of variables. It's always
possible to check a function's environment with func_name.__globals__.
- It's possible to make your own @decorators, which is just shorthand for a
function that wraps the decorated function.
- Lambda syntax is `lambda args : expression', e.g. `f = lambda x,y : x + y'.
- To have properties with accessors/mutators in classes, use the @property
decorator on a function by the name of the property, and add a
<prop_name>.setter or <prop_name>.deleter. There's another method that
involves the property() function.
- It's possible to dynamically import modules. This makes it possible to have
two modules with the same name for a function or class, and have the
implementation applied based on some conditional.
- Whenever any part of a package (a directory of modules) is imported, the
__init__.py file is executed. It's important that this contain at least a
declaration like `__all__ = ["module1", "module2"]' to handle * imports.
This is decent book for learning Python, and I probably should have just read
this one (or even just the official Python docs) and found some exercises
elsewhere.
*** CANCELED Spring Recipes: A Problem-Solution Approach (2nd Ed.)
- State "CANCELED" [2011-04-05 Tue 09:05]
The most recent Spring book.
Since my work's changed such that I no longer need this, canceling. I
certainly won't be reading up on this unless I have to, after all.
*** DONE GNU Emacs Manual
- State "DONE" [2011-04-11 Mon 18:55]
- State "STARTED" [2011-04-06 Wed 14:50]
Until now, I had only used this for a reference (as I'd already read a few
other Emacs books). Reading this from start to end (skipping the parts I
already know completely or for modes I don't use) and making notes of anything
I spot that will be useful in everyday life. Since I plan to use Emacs until
death, time spent learning more useful commands is never time wasted.
Online at http://www.gnu.org/software/emacs/manual/html_mono/emacs.html which
is the same as the Emacs info page (C-h i m Emacs).
Notes:
- To send a numeric argument to a command prefix with M-<digit>. To send a
multi-digit argument to a command, continue the numeric prefix while holding
Meta. To send a negative numeric command, use M-- (e.g. M-- 9 C-n scrolls up
9 lines). C-u can start and end a numeric prefix too.
- Prefix normal commands with C-u to multiply them by 4. Subsequent C-u
prefixes multiply further (e.g. C-u C-u C-n scrolls down 16 lines).
- Use C-u M-q to fill-paragraph and justify.
- C-x z calls repeat, which repeats the last command. Prefixing this with a
numeric argument can be rather handy. You can also run repeat and hit `z' as
many times as necessary to continue repeating.
- C-s and C-r work in the minibuffer for M-x history.
- Browse the E-Lisp libraries with C-h p.
- C-x C-x swaps point and mark and selects. Especially handy after C-<SPC>
C-<SPC> or C-y.
- Useful mark commands: M-@ (mark-word), M-h (mark-paragraph).
- Use M-m to jump to the first non-whitespace character in indented code
(easier than C-a TAB).
- Use the mark ring to get back to previous spots of interest in a file. Set a
mark point without activating it with C-<SPC> C-<SPC>. Then use C-u C-<SPC>
to jump to previous marks. I've setq-ed set-mark-command-repeat-pop to t in
~/.emacs in order to allow for mark ring traversal without popping them off
the stack.
- Use M-^ to join lines.
- C-S-backspace kills whole line. This is easier than C-a C-k C-k, but doesn't
work in terminal sessions inside urxvt (due to control-shift being captured).
Next time I recompile urxvt, flag configure with --disable-iso14755.
- View the contents of the kill-ring with C-h v kill-ring.
- Text accumulation functions can be handy if copying a bunch of regions to a
buffer, mostly with M-x append-to-buffer (along with several others).
- Rectangle region commands: C-x r k (kill-rectangle), C-x r y
(yank-rectangle), C-x r c (clear-rectangle), M-x string-insert-rectangle.
- Registers: These are single-letter named places to store content. They can
be useful for collecting or repeatedly yanking in a macro (and other uses).

- M-x visual-line-mode enables word-wrap (instead of the normal line-wrap).
- After searching, C-u C-<SPC> moves back to the pre-search point.
- M-e (forward-sentence), M-a (backward-sentence), M-} (forward-paragraph), M-{
(backward-paragraph), M-k (kill-sentence), C-x <DEL>
(backward-kill-sentence).
- M-r (move-to-window-line) with no args will move to the center of the window.
- C-h C-h b will show the active keymap.
- In isearch, use C-q to enter control characters (e.g. C-q C-j to add ^J to
the search string for line breaks). Use C-w and C-y to pull in additional
words or lines.
- Transposing: M-t (transpose-words), C-x C-t (transpose-lines).
- Spelling: M-$ (ispell-word).
- Use M-x auto-revert-tail-mode to tail -f a file by putting the cursor at the
end and enabling this mode. This updates every 5 seconds.
- M-x diff-backup will diff a file with its previous backups. Very handy.
- Use emerge for merging files. Commands are similar to ediff (a, b, n, q,
etc.). May want to set this up as git's merge tool later (added a task). I
think I still prefer ediff for more simple operations though.
- To edit over FTP, use C-x C-f /ftp:user@host:/path/to/file. The same syntax
works for other protocols.
- /: can also be used for quoted file names, e.g. /:/home/me/$5%~.txt.
- Window resizing commands: C-x { (enlarge-window-horizontally), C-x }
(shrink-window-horizontally), C-x ^ (enlarge-window), C-x +
(balance-windows), M-x shrink-window (maybe rebind C-x - to this).
- After using mouse-3 to select a region, a second mouse-3 in the same place
will kill the region.
- M-^ (delete indentation) will merge the current line to the previous and
delete the indentation (basically the opposite of C-j).
- M-o M-s (center-line).
- When filling paragraphs, use C-x . to set the fill prefix (usually, the
number of spaces that should precede each line), then running M-q. The fill
prefix is set to whatever is between the line beginning and the point though,
so it can be used for any kind of fill (e.g. program comments). Be sure to
clear it (C-a C-x .) if I don't want to keep it active in that buffer.
- Use M-x longlines-mode to create files with long lines that aren't saved with
newlines (only displaying soft newlines).
- C-h S (info-lookup-symbol) will scan the info files for the symbol searched
for. This is useful for languages with info pages installed, like elisp.
- To refresh the woman manpage scan (used for completion), send woman a numeric
argument.
- Use eldoc-mode when programming elisp to auto-display function argument
lists in the echo area.
- Use M-n and M-p in compilation buffers to navigate to errors without
selecting the source buffer. C-x ` and M-g p do the same but do select it.
Review these notes occasionally until I've internalized all of these new
commands in my standard Emacs command lexicon. I may even read this manual
again in a few years to pick up stuff I didn't find interesting this time (like
some of the advanced query commands, GUD, GTK customization, etc.).
*** DONE Hadoop: The Definitive Guide
- State "DONE" [2011-04-21 Thu 08:47]
- State "STARTED" [2011-04-03 Sun 00:31]
I may read Hadoop in Action after this and possibly Pro Hadoop after that
(though the latter is supposedly not that great). Apparently, I have an older
version of this book, missing the chapter on Hive, so I might pick up a hard
copy (as I'm going to be using this regularly), but I'll give the other books a
look first to see if they cover the missing stuff.
Notes:
- The org.apache.hadoop.io package contains the Hadoop types (optimizing for
network serialization). Use these types with generics in the
org.apache.hadoop.mapred package. Within my map() and reduce() functions,
convert these to Java base types to manipulate them and back to the Hadoop
types on the way out.
- Mapper/Reducer classes: Mapper<KeyIn, ValueIn, KeyOut, ValueOut>,
Reducer<KeyIn, ValueIn, KeyOut, ValueOut>. Mapper.Context and
Reducer.Context are passed into map() and reduce(). These contain various
context fields/methods and write output.
- Required components of a MapReduce program:
- A MyMapper class with a map() method.
- A MyReducer class with a reduce() method.
- A third class that runs the job.
- Combiner functions: Filters map() output locally on the data node. Can be
used to optimize amount of data xferred. Not applicable to certain kinds of
algorithms.
- Using Hadoop Streaming makes testing much easier, since you can just run it
outside of Hadoop to test.
- See pg. 55 for an example of testing the full workflow using Unix pipes.
- See pg. 56 for a streaming command line example.
- Maybe check out Dumbo (http://klbostee.github.com/dumbo/) for writing
MapReduce programs in Python.
- Run `hadoop fsck -blocks' to check cloud health.
- Namenodes can be backed up two ways, by writing persistent state to two
filesystems (one local, one NFS mounted) and by configuring a secondary
namenode as a failover (which lags a little, probably causing some data
loss).
- The 2nd column from the output of `hadoop fs -ls' is the replication count
(corresponds to dfs.replication in core-site.xml).
- Import org.apache.hadoop.dfs.MiniDFSCluster to create an in-process HDFS
cluster (for testing purposes).
- A distance metric (node, rack, datacenter) is used to determine latency of
requests instead of bandwith measurements.
- For HDFS reads, the HDFS client gets the first few block locations from the
namenode, then reads blocks directly from the data nodes. The DFSInputStream
(returned from FSDataInputStream) finds the next block internally. See
pg. 63 for full details.
- For HDFS writes, the HDFS client creates the new file entry on the namenode
(blocks not yet assigned), then the DFSOutputStream (returned from
FSDataOutputStream) splits the data into packets andwrites then to the
internal data queue, which is consumed by the DataStreamer (which asks the
namenode to allocate blocks). DataStreamer streams the packet to the
datanode, and if replication is enabled, that node will pass it along after
writing to the next node. After this completes successfully, an ack is sent
back to the DataStreamer, whence it is removed from its internal ack queue.
- To ensure data is actually written to HDFS, either an explicit sync() must be
called, or close() (which performs an implicit sync()).
- For parallel copying of files in HDFS, used distcp (a MapReduce job). An
example use case is parallel copying between two clusters.
- Use the balancer to rebalance a cluster.
- Use HAR (Hadoop Archive) files to collect many smaller files (as small files
eat up namenode space). Collect files with `hadoop archive -archiveName
files.har /my/files /user/hadoop'. This will create a files.har in
/user/hadoop. Look at the contents of the HAR with `hadoop fs -lsr
har:///user/hadoop/files.har' or `hadoop fs -lsr
har://hdfs-localhost:8020/user/hadoop/files.har'. Use -rmr (recursive rm) to
delete HAR files, since they're stored considered directories by Hadoop. HAR
files cannot be updated.
- Disable checksums (probably should never do this, due to the numerous failure
points) by passing false to FileSystem.setVerifyChecksum();
- Use the various Hadoop types derived from the Writable interface to write
various data, providing it with the serialization Hadoop expects. See
starting at pg. 110 for complete info.
- When serializing, always use the Hadoop serialization
(e.g. org.apache.hadoop.io.serializer.WritableSerialization).
- File-based data structures: SequenceFile and MapFile are two specialized data
structures. A SequenceFile can be handy for, say, a log file that includes
binary data, but it's main benefit is as a way to collect many small files.
A MapFile is the same, but sorted to permit lookups. For SequenceFiles or
MapFiles with binary data, use SequenceFileInputFormat (such as its variant,
SequenceFileBinaryInputFormat).
- MapReduce writing workflow: Write map and reduce functions with unit tests.
Write the driver program to manage the job, including the ability to run it
from the IDE with test data (thus making it debuggable). If the controller
fails, expand the unit tests to cover the cause. Then try it on the cluster.
When non-obvious errors occur here, debug them using something like
IsolationRunner. It's possible to attach a debugger to Hadoop as well (look
into this). Finally, attempt to make the tasks run faster, perhaps by using
task profiling (some of which Hadoop provides).
- A Configuration can be read from an XML resource file. See pg. 116 for the
structure. The defaults for the system are defined in core-default.xml
(contained internally), the defaults for the site are in core-site.xml.
These conf files can be used for switching between clusters when running
projects. A good practice is to make a hadoop-local.xml (local FS and
jobtracker), hadoop-localhost (namenode and jobtracker running on localhost),
and hadoop-cluster.xml (actual addresses). Use these from the CLI with
hadoop's -conf flag. Use -D on the CLI to change these properties
individually (include a space, unlike -D for the JVM). The conf file can
even include the map and reduce classes.
- IDE setup: Unpack the distribution, then add the distro's lib directory to
the classpath. For Eclipse, there's a Hadoop plugin for MapReduce/HDFS
integration at: http://wiki.apache.org/hadoop/EclipsePlugIn
- This book suggests Mockito for unit tests, but it doesn't seem necessary.
- Running jobs: A command like `hadoop jar job.jar pkg.MainClass -conf
conf/hadoop-localhost.xml input/all.csv output.txt'. In this example, if
job.jar had the main class defined in the manifest, it wouldn't need the
pkg.MainClass parameter.
- The jobtracker page (http://localhost:50030/jobtracker.jsp) includes a full
history of up to 100 jobs (configurable by the
mapred.jobtracker.completeuser.job.maximum property).
- Chukwa is a Hadoop log analyzer.
- Use Reporter.setStatus() to set the task status in the jobtracker page.
- See the various logs in $HADOOP_HOME/logs for various types of logging.
logs/userlogs have the logs for specific attempts.
- Remote debugging: Set the configuration property keep.failed.task.files to
true. This has the tasktracker keep enough info to be able to rerun on the
same input data. Run the job again and note which node the task fails on and
the ID (attempt_*) using the web UI. Then use IsolationRunner with those
retained files as input (saved to mapred.local.dir, under
taskTracker/jobcache/job-ID/task-attempt-ID). Then allow the remote debugger
to connect using `export
HADOOP_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000'.
Suspend makes it wait until the debugged is connected. Now re-launch the
task with `hadoop org.apache.hadoop.mapred.IsolationRunner ../job.xml'. To
keep task data for non-failing tasks, edit keep.task.files.pattern to a regex
that will match the IDs of the tasks I want to keep.
- See table 5-3 for a tuning checklist.
- A convenience method to start jobs is JobClient.runJob(conf). This calls
submitJob().
- The TaskRunner (an instance of which is created by the tasktracker) starts a
new JVM to run each map or reduce task, allowing for no interference with its
own. It's possible to reuse JVMs between tasks though.
- Jobs and tasks each have a status, which can be checked to determine the
state (running, successfully completed, failed), progress, counters, status
message, etc. For example, JobClient.runJob() returns a RunningJob instance,
which has various status methods, like mapProgress() and reduceProgress().
- For site configuration, look at the various -default.xml files in the src.
These have the full configuration options.
- Hadoop has a choice between the FIFO scheduler (the default) and the Fair
Scheduler, which is more suited for a multi-user environment.
- If map functions are outputting a lot of data, it may make sense to enable
compression on it to lower the amount of data to transfer to the reducer.
Set mapred.compress.map.output to true to do this.
- Enable skipping mode (via the SkipBadRecords class), if it's not possible to
skip them (due to some 3rd-party library). Ideally, these should be filtered
in the map function.
- In a Python script, grab environmental variables with
`os.environ["mapred_job_id"]'. Note that the "." characters are converted to
underscores.
- Function types:
map: (K1, V1) -> list(K2, V2)
combine: (K2, list(V2)) -> list(K2, V2)
reduce: (K2, list(V2)) -> list(K3, V3)
partition: (K2, V2) -> integer
- The default for map and reduce is to do nothing. This can be useful if it's
just necessary to do one of either.
- The default seperator between keys and values is a tab. This can be changed
for both the mapper and reducer, on both input and output.
- To disable file splitting (potentially necessary on some map functions, like
ones that sort the contents) subclass FileInputFormat and override
isSplitable() to return false. See other similar customizations on pg. 185.
- TextInputFormat is the default InputFormat. It takes text file input and
considers each line the value, with the key being an integer representing the
offset. KeyValueTextInputFormat is slightly more useful, as it assumes the
key is at the beginning of the line, separated from the value text by a
delimiter (tab being the default).
- If long tasks are timing out (they're normally expected to report progress
every 10min or they'll be killed), just write status messages on it.
- List Hadoop jobs from the CLI with `hadoop job -list'.
- Use DistributedCache to disseminate jars/files to each node at job submission
time.
- MapReduce provides a bunch of built-in features, such as counters (for input
records, output records, records skipped, and many other things), sorting
(via the sort stage, which can be programmed to, including in streaming with
Python), joins (though Hive is easier for this), the DistributedCache (with
the -files flag on the job call. See example for accessing datafiles
starting on pg. 239), and various MapReduce library classes.
- Apache ZooKeeper: A "distributed coordination service". See notes under
Apache ZooKeeper task.
- If Hadoop gets stuck in safe mode forever, try `hadoop dfsadmin -safemode
leave'.
Skipping Hadoop setup/administration, Pig, HBase, and Cloudera, for now.
*** DONE Weka Manual 3.7.3
- State "DONE" [2011-05-09 Mon 08:49]
- State "STARTED" [2011-05-06 Fri 07:14]
Notes:
- Weka assumes input data is a flat file or relation, each data point being
represented by a fixed number of attributes. Weka can also use as input the
results from a database query.
- Datasets, the fundamental machine learning data structure -- typically a
2-dimensional array, can contain these attributes in Weka: nominal
(predefined values), numeric (real, integer, numeric), string (enclosed in
double quotes), date, and relational.
- ARFF syntax: Created a test.arff file, added to GitHub.
- See examples (on GNU/Linux) in /usr/share/doc/weka/examples.
- To validate and run some basic stats on an ARFF file, run `java
weka.core.Instances soybean.arff'.
- To import a CSV file into an ARFF file automagically, run `java
weka.core.converters.CSVLoader data.csv > data.arff'.
- Send -h to any Weka class to get its command line parameters.
- A classifier is a mapping from all-but-one dataset attribute to the class
attribute, specifics varying between classifiers.
- Folds: These provide an alternative training method vs. independent
training/test sets. The data is split into n folds and one is randomly
chosen for testing, while the other n - 1 folds are used for training.
Results are then averaged over all folds.
- Use the weka.filters package to add/remove attributes, in supervised and
unsupervised form. weka.filters.supervised is for supervised filtering,
which can take advantage of class information (default behavior with `-c
last'). Example: java weka.filters.unsupervised.instance.Resample -i
data/soybean.arff -o soybean-5%.arff -Z 5
- For classifiers, if `-T test_file' is left out, a cross-validation will occur
on the input data. This file needs to redefine the attributes.
- Confusion matrix (contingency table): Correctly classified instances are the
sum of the diagonals.
- For classifiers, -i is useful, since it shows the TP (true positive), FP Rate
(false positive rate), etc.
- For big jobs, use -Xmx1024m to ensure the task has enough memory.
- If I have a bunch of classifier output in .out files, having run stuff like
`java weka.classifiers.trees.J48 -t soybean.arff -x 5 -i -k > test.out', I
can run something like `cat test.out | grep -A 3 "Stratified" | grep
"^Correctly"'. Doing this on all my .out files will give me a list from
which I can tell which classifier is the best.
- GUI tools:
- Explorer: Standard Weka tool.
- Experimenter: Tool for testing between different learning schemes.
- KnowledgeFlow: Same as Explorer, but with drag-and-drop.
- SimpleCLI: CLI for systems like Windows.
- ArffViewer: Tool for viewing ARFF files in table view.
- To save an image while looking at it from the Visualize tab, use
S-M-<mouse-1>. If in Gnome, disable the movement key in
System->Preferences->Window.
- Right click on entries in the Result List for various options, like
visualizing the results.
- In scatter plots, jitter introduces random displacement on points, possibly
allowing for concentrations to be more obvious (like if many points are at
the same spot).
- Declare date attributes with `@ATTRIBUTE timestamp DATE "yyyy-MM-dd
HH:mm:ss"'. In the data, this attribute will then look something like
`"2001-04-03 12:12:12"'.
Skipped chapters on Experimenter, KnowledgeFlow, Bayesian network classifiers,
and the Appendix. May read these later (particularly the Bayesian network
classifiers chapter).
*** CANCELED IEEE Std 1003.1-2008
- State "CANCELED" [2011-05-25 Wed 12:33]
The POSIX standard. Might be worth a read.
http://pubs.opengroup.org/onlinepubs/9699919799/
Read some, but it's not useful information in my life, nor even remotely
interesting.
*** DONE Stuxnet Under the Microscope
- State "DONE" [2011-05-29 Sun 02:43]
- State "STARTED" [2011-05-29 Sun 01:15]
Already glanced at the decompiled Stuxnet code which definitely supports the
supposition that it was written by a professional team of developers using a
iterative development process with extensive debugging/testing. I've seen
plenty of other amateur exploit code, and this looks more like something I'd
see at work. This professional analysis might have made some interesting
additional observations.
TMP file note: I don't buy the suggestion (made by Cryptome) that "myrtus"
means "My RTUs", as programmers don't structure their directories the way it's
used in that path. This implies that it's either a username, project code name
(my guess), or a red herring. If not the latter, then it's probably of Israeli
origin. It's almost definitely a product of some Western nation-state, since
the literals used therein are the type seen by English-speakers. However,
references to "futbol" imply non-US origin, though everyone in the US knows
about the rest of the world's soccer fixation.
Pretty interesting program, but this paper doesn't really do that great of a
job describing it. Generally, I'm not interested in reading post-mortems like
these, particularly not on Windows, but made an exception here, due to this
being the most advanced targeted attack to date and having implications related
to my field of work.
Update: The NYT supposedly confirmed that Stuxnet was a join US/Israeli
operation, with development in the US and config/deployment by Israel.
*** DONE Epigrams on Programming (paper)
- State "DONE" [2011-06-17 Fri 06:45]
- State "STARTED" [2011-06-17 Fri 05:23]
Some learned wisdom here, though I semi-disagree with the Lisp criticisms.
My favorites:
19. A language that doesn't affect the way you think about programming, is not
worth knowing.
34. The string is a stark data structure and everywhere it is passed there is
much duplication of process. It is a perfect vehicle for hiding
information.
39. Re graphics: A picture is worth 10K words - but only those to describe the
picture. Hardly any sets of 10K words can be adequately described with
pictures.
40. Some programming languages manage to absorb change, but withstand progress.
54. Beware of the Turing tar-pit in which everything is possible but nothing of
interest is easy.
93. When someone says "I want a programming language in which I need only say
what I wish done," give him a lollipop.
94. Interfaces keep things tidy, but don't accelerate growth: Functions do.
95. Don't have good ideas if you aren't willing to be responsible for them.
*** DONE Tutorial on Good Lisp Programming Style (paper)
- State "DONE" [2011-10-26 Wed 13:31]
- State "STARTED" [2011-10-28 Fri 11:48]
Large series of postscript slides by Peter Norvig. Grabbed a PDF copy for
reading elsewhere.
** Books: Math
*** DONE Common Mistakes in Using Statistics (course)
- State "DONE" [2011-06-01 Wed 06:10]
- State "STARTED" [2011-06-01 Wed 04:59]
A coworker attended this class and recommends the material. Slides in a file
called common_mistakes_in_statistics-2011.pdf.
Notes:
- The file drawer problem: The problem that studies who are unable to show
findings that reject the null hypothesis languish unnoticed, while the ones
that do not attract attention.
- An "event" is the thing the probability of is being measured.
- P(A), the probability of event A, is defined as m/n, where A is satisfied by
exactly m of n outcomes.
- P(E | C) is the probability of event E, given condition C.
- A random (stochastic) process is a process where the outcome is
probabilistic rather than deterministic.
- A random variable is a variable that depends on a random process,
e.g. tossing a die is a random process and the number on the top is the value
of the random variable.
- Bias is systemic if it favors an outcome over another, e.g. telephone surveys
that miss various types of people, like those who don't want to participate
(this is a type of sample called a voluntary response sample).
- Extrapolation is a common error where conclusions are drawn from beyond the
range of data.
- Standard deviation provides no asymmetry information on skewed distributions.
Many standard statistical techniques are invalid for these.
More info at: http://www.ma.utexas.edu/users/mks/statmistakes/TOC.html
*** DONE Bayesian Statistics for Social Sciences (course)
- State "DONE" [2011-06-02 Thu 09:52]
- State "STARTED" [2011-06-01 Wed 06:45]
Another statistics course taken by coworker. Slides in a file called
bayesian_statistics-2011.pdf.
Notes:
- Non-Bayesian statistical methods include method of moments and maximum
likelihood.
- Basic setup:
- Two events of interest: A and B.
- E.g., A is a hypothesis about some phenomenon and B is observed data.
- We may know P(B|A), which is the probability of the data given different
parameter values.
- We want to know P(A|B), which is the probability of the hypothesis being
true given the data.
- Bayes' Theorem: P(A|B) = (P(B|A)P(A))/P(B).
- IID = independently and identically distributed. E.g., roulette wheel spins
are IID.
These slides are a bit hard to follow and annoying to read (since it's a
PowerPoint export). Maybe they'd make more sense with the accompanying
lecture. Giving up on these about 1/3rd in and will find a different source
for this information.
** Books: General Non-fiction
*** DONE The Lie Behind the Lie Detector
- State "DONE" [2011-02-01 Tue 00:55]
- State "STARTED" [2011-01-31 Mon 22:20]
Super useful. Luckily for me, I had most of this figured out on my own, but
now that I know the rest of this, it'll be a breeze from now on.
*** DONE Assassination Politics
- State "DONE" [2011-02-22 Tue 02:02]
- State "STARTED" [2011-02-22 Tue 01:01]
I think what's still missing here is the digital currency. If an existing or a
new one gains real traction, then that will be the final piece. Either way,
it's probably inevitable (though unlikely in my lifetime). If implemented, the
global economy would really resemble a healthy macro-organism (at least to the
extent possible given average human intellect), which purges itself of viruses.
*** DONE Summa Logicae
- State "DONE" [2011-02-22 Tue 02:47]
- State "STARTED" [2011-02-22 Tue 02:29]
Give this partial translation/commentary of Book I a read:
http://www.iupui.edu/~peirce/writings/v2/w2/w2_32/v2_32.htm
Like a lot of early philosophical work, this suffers from the lack of language
available to effectively communicate the points. Some tend to be banal and/or
needlessly expanded upon, but that could also be failings of the author. In
any case, this particular presentation of it sucks and reads like someone's
book report, though it's by Charles Pierce (himself a renown philosopher and
mathematician). I've seen enough to know when I'm wasting my time.
*** DONE Automatic Wealth: The Six Steps to Financial Independence
- State "DONE" [2011-04-05 Tue 02:56]
- State "STARTED" [2011-04-04 Mon 15:10]
This book bears all the signs of being mall-bookstore fodder, but I'll give it
a few hours. If this guy knows something I don't, then it'll be worth the
time.

- "You need a lot of money to retire." A 65 year old woman in here can't
retire because she "only" has $600k.
- The author thinks the stock market "historically averages a 10% return".

- Very repetitive. For such a short book, this is pretty lame.
There is some good advice in here, like not including cars and houses in net
worth, but the overall plan is a stupid and inefficient one and the end
scenario in mind here is a glut of consumerism.
*** DONE The Millionaire Next Door
- State "DONE" [2011-04-09 Sat 02:28]
- State "STARTED" [2011-04-07 Thu 13:03]
The classic personal finance text about getting rich via mild frugality and a
lifetime of work. This isn't my plan, but advice from this book has
occasionally made its way in front of my eyeballs and what I've seen I
generally approve of, so I might pick up a few ideas from a full read.
Pretty good, but didn't learn anything. There's way too many anecdotes, but
for a financial book, it does contain a lot of facts and, more importantly,
original research. However, a lot of these facts are considerably outdated.
The central message, that self-made millionaires' lifestyle is the opposite of
popular conception, is something I've already long since known.
*** DONE Transgressing the Boundaries
- State "DONE" [2011-04-11 Mon 14:18]
- State "STARTED" [2011-04-11 Mon 13:37]
The Sokal Hoax paper with the full title Transgressing the Boundaries: Towards
a Transformative Hermeneutics of Quantum Gravity. Already read this three
times since 1998, but it never fails to be entertaining.
*** DONE The Abolition of Work
- State "DONE" [2011-04-15 Fri 22:51]
- State "STARTED" [2011-04-15 Fri 10:04]
I read this once before, but it was around 1995 and I was much more stupid
then.
http://www.zpub.com/notes/black-work.html
While I agree with the dislike of having a job and it (at least, with some
people) being a rational goal to eliminate it, this essay is obviously written
by a person with rather muddled thoughts (quoting Foucault is always a dead
give-away), writing to impress more than to inform (or maybe he's just not
smart enough to write any other way). In any case, I didn't see much of a
factual basis for the argument here and I'm still waiting for innumerable
issues to be addressed by proponents of this philosophy, as I've thought about
it and can't resolve the majority of them.
I'd also quibble with the definitions here. I'd define work, in the
non-physics sense, as something like "expending energy to produce a desired
output", which I don't have any particular problem with. If you desire the
output and are willing to expend the energy to get it, then this transfer is
something you want.   Were I to write a treatise on the subject, it would focus entirely
upon identifying the real cost/benefits of the transaction, at least where its
optional (non-optional employment is a different topic).
Finally (though there's many other problems here), I'd argue that for most
humans, they already have too much play time, and that, especially in current
Western consumerist culture, the play time they do engage is destructive, both
to themselves and externally.
Basically, the issue is too multi-faceted to be waved away with a social
commentary.
*** DONE Syndicate Wars Official Guide
- State "DONE" [2011-04-17 Sun 12:54]
- State "STARTED" [2011-04-17 Sun 03:41]
Available here:
http://syndicate.lubie.org/swars/html/swars_manual_official_guide.php
Didn't learn anything.
*** DONE FM 3-07
- State "DONE" [2011-05-15 Sun 00:01]
- State "STARTED" [2011-04-19 Tue 08:12]
Reading for some work-related context. Skipping the legal and counter-drug
content herein.
Not very interesting. Too generalized, poorly focused, and administrative for
me to learn much from. Quit about 1/3 in.
*** DONE Fixing Intel (paper)
- State "DONE" [2011-05-15 Sun 00:01]
- State "STARTED" [2011-05-15 Sun 00:55]
Full title of Fixing Intel: A Blueprint for Making Intelligence Relevant in
Afghanistan.
Not a bad read, considering my lack of interest in the topic. I'm inclined to
agree with the paper's recommendations, but I can't say for sure, as I don't
really know the problem space enough to decide with any authority.
*** DONE The Logic of Violence in Civil War
- State "DONE" [2011-05-15 Sun 16:25]
- State "STARTED" [2011-05-15 Sun 15:10]
Written by someone who obviously isn't familiar with the wonders of operating a
spell checker. However, this is a good summary of the cost/benefit analysis
performed by parties involved in civil war.
*** DONE Institute for the Study of War papers
- State "DONE" [2011-05-24 Tue 05:00]
- State "STARTED" [2011-05-16 Mon 04:56]
Lots of good papers on this site. Just reading some of the recent ones.
http://www.understandingwar.org/afghanistan-project/publications/reports
Papers read:
- Defining Success in Afghanistan: Highly informative and a great overview of
the situation in early 2011.
- Reversing the Northeastern Insurgency: Useful info on the situation in
Kunduz, Baghlan, and Takhar.
- Haqqani Network in Kurram: Useful info on a major Sunni terrorist target in
FATA.
- Counterinsurgency in Helmand: At the time of this writing, Helmand was way
better than a year ago, but still has some serious issues. Progress has
continued since then, but it won't be till late summer until we know if it's
a trend shift.
- Operation Moshtarak - Taking and Holding Marjah: A bit outdated, but this
battle was the decisive action of the operation on an enemy stronghold.
Reading these definitely helped a lot. They tend to be a bit positively
focused, but that could be because things are a little better now than a year
ago. I'll check back here in 6 months to see if any new stuff is posted.
*** DONE The Takhar Attack (paper)
- State "DONE" [2011-05-25 Wed 08:58]
- State "STARTED" [2011-05-24 Tue 07:44]
Not particularly interested in the specific incident, but it's a good case
study. Recommended by coworker. From: http://aan-afghanistan.com/
This is a bit more pessimistic of an analysis than some articles I read on this
incident. But from my position without any source data, seems the more likely
and is a pretty big, and all too typical, screw-up. Skipped legal stuff.
*** DONE Tableau 6.0 manual
- State "DONE" [2011-05-25 Wed 09:00]
- State "STARTED" [2011-05-24 Tue 06:23]
Tableau is just a boring visualization software package, so just need to skim
this.
Predictably boring, but it is a manual. I'll keep this handy for whenever
using it though to lazy-load the ability to do stuff with it.
*** DONE Agalmics: The Marginalization of Scarcity
- State "DONE" [2011-07-21 Thu 14:11]
- State "STARTED" [2011-07-21 Thu 13:59]
Article by the now deceased founder of Freenode, Bob Levin, on free and open
source software.
Hopelessly naïve. Ignores inherent properties of physical resources. For
example, a population of beings trapped on a single planet have a fixed amount
of land resource. As the population increases, land becomes increasingly
scarce. This essay is about copyrights though, but the basis for the argument
is flawed.
*** DONE Why Socialism is "Impossible" (paper)
- State "DONE" [2011-08-10 Wed 21:48]
- State "STARTED" [2011-08-10 Wed 17:28]
Okay for a summary of the Von Mises assertion along these lines. Didn't learn
anything new though.
*** DONE Ghost in the Wires
- State "DONE" [2011-11-10 Thu 13:54]
- State "STARTED" [2011-11-08 Tue 23:41]
Kevin Mitnick's most recent autobiography. An okay read. He definitely did
some interesting stuff, but more technical details would've made it a better
use of a day. I really only know his life story now, and almost nothing new
about anything else.
*** DONE In the Beginning...Was the Command Line
- State "DONE" [2011-11-11 Fri 13:53]
- State "STARTED" [2011-11-11 Fri 12:44]
Well written at times, boring at others. Apart from occasionally being
entertained by a witty phrase or two, not much was learned here. Essays like
these are common on HN, after all, so I've heard it all before. Much of this
is similar to my perspective on CLIs and technology culture, but I think
Stephenson's techno-epistemology has a few holes in it that mine doesn't.
Regarding the technical content, it's shallow, outdated, and even contains
minor errors (e.g. C++ is not a dialect of C, not all GNU/Linux distros have
similar directory structures, Emacs isn't spelled with a lower case first
letter, etc.). I might've enjoyed this as a young n00blet, had I not been
already well past that point by the time this was published.
** Books: Fiction
*** DONE Lucifer's Hammer
- State "DONE" [2011-01-21 Fri 17:12]
- State "STARTED" [2011-01-01 Sat 16:13]
A decent post-apocalypse novel, but only by comparison to others in its genre.
*** DONE Blindsight
- State "DONE" [2011-01-24 Mon 21:57]
- State "STARTED" [2011-01-23 Sun 22:56]
Online at: http://www.rifters.com/real/Blindsight.htm
A truly excellent book, among the top of hard scifi novels I've ever read. An
excellent exploration of sentience, a realistic portrayal of human thought,
good writing, realistic characters, realistic progression of future technology
(minus one point), and one of the better and more interesting ETIs out there.
Not totally perfect, but more than what anyone could expect from a novel.
*** DONE The Big Bow Mystery
- State "DONE" [2011-01-29 Sat 19:54]
- State "STARTED" [2011-01-27 Thu 20:12]
Supposedly the first well-done "locked room mystery".
http://www.gutenberg.org/cache/epub/28164/pg28164.txt
It's okay, but nothing special. IMO, a mystery should be solvable by the
reader if he's smart enough. The final exposition should either confirm or
deny your theory then. This isn't one of those though.
*** DONE Rogue Farm
- State "DONE" [2011-01-29 Sat 20:36]
- State "STARTED" [2011-01-29 Sat 20:03]
http://www.bestsf.net/presents/RogueFarm.html
Nothing interesting, due to being too short for anything to happen with the
ideas here.
*** DONE A Colder War
- State "DONE" [2011-01-30 Sun 18:00]
- State "STARTED" [2011-01-30 Sun 01:39]
A Cthulhu mythos/cold war crossover. OK, but not all that great.
*** DONE The Murders in the Rue Morgue
- State "DONE" [2011-02-06 Sun 02:50]
- State "STARTED" [2011-02-06 Sun 01:44]
http://poestories.com/text.php?file=murders
Not that great.
*** DONE Thirteen (Black Man)
- State "DONE" [2011-02-06 Sun 08:34]
- State "STARTED" [2010-11-09 Tue 22:28]
Dragged out, with an uninteresting underlying plot. It's basically just a
standard catch the bad guy drama that might just as well not be scifi.
Finishing this turned into a real chore. Finally giving up about 80% in.
*** DONE Singularity Sky
- State "DONE" [2011-02-07 Mon 21:09]
- State "STARTED" [2011-01-31 Mon 07:33]
Has some okay parts and is less ridiculous than Accelerando (same author), but
still suffers from some of the same problems.
*** DONE The Last Answer
- State "DONE" [2011-02-15 Tue 01:17]
- State "STARTED" [2011-02-15 Tue 01:03]
Okay, for a short story.
*** DONE Nemesis
- State "DONE" [2011-02-18 Fri 00:14]
- State "STARTED" [2011-02-15 Tue 01:24]
Asimov novel. Competently written, but short of really good, due to a lot of
the content being a bit too generic and done before.
*** DONE Tlön, Uqbar, Orbis Tertius
- State "DONE" [2011-02-20 Sun 01:14]
- State "STARTED" [2011-02-19 Sat 15:06]
Disappointing given its supposed significance.
*** DONE The Hound of the Baskervilles
- State "DONE" [2011-02-27 Sun 12:34]
- State "STARTED" [2011-02-24 Thu 23:03]
http://www.gutenberg.org/files/3070/3070-h/3070-h.htm
Good enough for a quick read.
*** DONE Thus Spake Zarathustra
- State "DONE" [2011-03-02 Wed 22:17]
- State "STARTED" [2011-02-22 Tue 03:17]
http://www.gutenberg.org/files/1998/1998-h/1998-h.htm
Nonstop baseless declarations and needless obscurantism. Giving up about 25%
in, having seen nothing of any substance yet.
*** DONE The Caves of Steel
- State "DONE" [2011-03-05 Sat 22:05]
- State "STARTED" [2011-03-02 Wed 22:20]
Pretty good for a detective novel, just average for a scifi novel.
*** DONE Foundation
- State "DONE" [2011-03-07 Mon 19:12]
- State "STARTED" [2011-03-05 Sat 23:13]
Lots of anachronisms, but the first half is okay despite that. Loses some of
its momentum later though. Ends up being just okay overall.
*** DONE The Naked Sun
- State "DONE" [2011-03-08 Tue 10:19]
- State "STARTED" [2011-03-07 Mon 19:14]
Better than The Caves of Steel.
*** DONE The Robots of Dawn
- State "DONE" [2011-03-13 Sun 22:51]
- State "STARTED" [2011-03-08 Tue 23:07]
About on par with the previous, but loses a few points for all the low brow
content.
*** DONE Robots and Empire
- State "DONE" [2011-03-22 Tue 21:47]
- State "STARTED" [2011-03-13 Sun 23:07]
Not that great. Fulfills it's mission of tying together the Robot and the
Foundation series, but I personally don't give a crap if they exist in the same
universe or not.
*** DONE Semper Mars
- State "DONE" [2011-03-27 Sun 04:11]
- State "STARTED" [2011-03-22 Tue 22:00]
First in the Heritage Trilogy. Might read the others if this is any good.
There's then 2 other trilogies after this in the same universe.
Garbage. Finished this one, but not reading the others. Books that glorify
military culture tend to be pretty stupid, and this is no exception.
*** DONE The Gods Themselves
- State "DONE" [2011-03-31 Thu 22:14]
- State "STARTED" [2011-03-27 Sun 19:24]
A stand-alone Asmiov novel. Pretty good sci-fi in the 1st and 3rd parts, but
the 2nd kinda sucked, forcing me to skip it and read a summary of it on
Wikipedia instead (to the effect of me feeling like I didn't miss anything).
*** DONE Araby
- State "DONE" [2011-04-01 Fri 15:12]
- State "STARTED" [2011-04-01 Fri 14:44]
A James Joyce short story. Reading this to sample the author's writing prior
to committing to reading Ulysses.
http://www.classicreader.com/book/344/1/
This is actually surprisingly good, though the theme isn't all that
interesting.
*** DONE The Fall of the House of Usher
- State "DONE" [2011-04-02 Sat 02:19]
- State "STARTED" [2011-04-02 Sat 01:29]
An Edgar Allen Poe short story.
Not that great.
*** DONE The Cask of Amontillado
- State "DONE" [2011-04-02 Sat 11:23]
- State "STARTED" [2011-04-02 Sat 02:20]
Another Edgar Allen Poe short story, involving revenge and immurement.
Not that great either.
*** DONE Tau Zero
- State "DONE" [2011-05-02 Mon 00:51]
- State "STARTED" [2011-04-27 Wed 12:32]
Supposedly a classic hard scifi novel.
Pretty good, but too much character development for my tastes. Much of the
purpose of the novel hinges on how human psychology reacts to the kind of
situations here, and that, at least for me, isn't all that interesting. Apart
from that, however, it's a good read.
*** DONE Assemblers of Infinity
- State "DONE" [2011-05-07 Sat 17:57]
- State "STARTED" [2011-05-04 Wed 17:12]
Takes place too soon in the future (11 years from 1993). Bland, uninteresting
characters. Plot meanders and never really does anything interesting.
Overall, an amateur effort.
*** DONE Solaris
- State "DONE" [2011-05-08 Sun 00:26]
- State "STARTED" [2011-05-07 Sat 18:12]
A classic hard scifi novel. Was put off from reading this for awhile due to
the crappy 2002 film based on it, but the book is actually extremely good and
well written.
*** DONE The Cold Equations
- State "DONE" [2011-05-08 Sun 09:11]
- State "STARTED" [2011-05-08 Sun 08:33]
Hard scifi short story from 1954. One of the stories that helped part of the
genre escape from its adventurism roots. An okay read, but nothing great.
*** DONE His Master's Voice
- State "DONE" [2011-05-14 Sat 19:26]
- State "STARTED" [2011-05-08 Sun 00:57]
While I'd probably more enjoyed these ideas here presented in non-fiction form,
this is, in a lot of ways, real science fiction should be, as apart from the
plot, it's almost not even fiction at all. The writing here is often top
quality, and I often found myself quite amazed by the expertly-crafted prose
(far beyond my writing ability).
*** DONE The Invincible
- State "DONE" [2011-05-19 Thu 07:27]
- State "STARTED" [2011-05-16 Mon 06:02]
Another Stanislaw Lem novel. This one isn't that great though.
*** DONE The Algebraist
- State "DONE" [2011-05-21 Sat 21:50]
- State "STARTED" [2011-05-21 Sat 20:48]
Giving Iain M. Banks another try. This book's the beginning of a new series,
so maybe it's less hammy than his proliferous Culture series. Going in with an
appropriate setting for the give-up threshold.
Bleh. Read enough to know this book isn't for me.
*** DONE Starfarers
- State "DONE" [2011-05-29 Sun 09:30]
- State "STARTED" [2011-05-21 Sat 21:58]
Another Poul Anderson (author of Tau Zero) novel.
Too much boring interpersonal relations content (often at the expense of the
scifi content). Then the scifi content itself is pretty generic and
uninteresting on top of that. There's also a lot of pointless filler with
side-stories that have absolutely nothing to do with the plot, which itself
meanders aimlessly. A lot of this is probably due to the way it was originally
published as a collection of short stories over many years, but I've seen that
done well too, so it's no excuse. A shame, since the scenario here could have
been used for great things.
*** DONE Out of the Dark
- State "DONE" [2011-08-04 Thu 21:13]
- State "STARTED" [2011-08-01 Mon 02:17]
A new alien invasion novel from 2010.
Also, the next time I'm reading a book and I encounter aliens that are just
cheap carry-overs of Earth animals, I'm insta-quitting.
*** DONE The Academy series
- State "DONE" [2011-08-13 Sat 23:56]
- State "STARTED" [2011-07-10 Sun 15:40]
Attempting to read all of these.
- The Engines of God: First book of The Academy series. Not bad, but has a lot
of long-winded filler.
- Deepsix: Pretty uncreative. The alien fauna/flora are all unimaginative
variants of Earth creature. Uses a serious physical impossibility of welding
carbon nanotubes. The plot line isn't just one long rescue/survival mission
and not very scifi.
- Chindi: A boring, predictable rescue-adventure with cartoon-like characters.

- Odyssey: Skipping this one.
- Cauldron: Same problems as the rest of the series. The final resolution of
the main issue across the series is thoroughly disappointing.
This author exhibits a large lack of imagination and seems content with
producing reams of text describing mundane and inconsequential activities.
Definitely not reading any more of these.
*** DONE The Cassini Division
- State "DONE" [2011-08-16 Tue 10:06]
- State "STARTED" [2011-06-19 Sun 22:25]
Parts of this are very well written, others not so much. The well written
parts are sometimes very good though. The main immersion-killer is all of the
socialist utopianism.
*** DONE Childhood's End
- State "DONE" [2011-08-23 Tue 09:45]
An earlier Arthur C. Clarke novel. Read this on the plane to Dubai.
Definitely the weakest novel by this author I've read so far.
*** DONE Library of the World's Best Mystery and Detective Stories
- State "DONE" [2011-10-01 Sat 14:48]
- State "STARTED" [2011-08-25 Thu 22:42]
Just grabbed a copy of this off Gutenberg for reading on an airplane trip.
The selections here are almost all pretty bad.
*** DONE Snow Crash
- State "DONE" [2011-11-30 Wed 16:18]
- State "STARTED" [2011-11-26 Sat 23:52]
Apart from all the lame pop culture (ninjas, skateboards, etc.), this book's
main problem is the thoroughly implausible premise.
*** DONE Iron Sunrise
- State "DONE" [2011-12-11 Sun 10:16]
- State "STARTED" [2011-12-01 Thu 14:34]
Better than the previous book in this series (Singularity Sky), but still not
that great.
** Technology and software
*** DONE bitcoin
- State "DONE" [2011-02-10 Thu 08:49]
- State "STARTED" [2011-02-09 Wed 15:30]
Installing this on Ubuntu so they can be generated on work power.
Setup:
- sudo apt-add-repository ppa:stretch/bitcoin
- sudo apt-get update
- sudo apt-get install bitcoin
- Run `bitcoin' and limit coin generation to one processor (since I need to get
work done too).
- Make note of bitcoin address.
Unfortunately, there's only that lame GTK client. I like the concept of
bitcoin though, so hopefully it'll catch on. Generating coins takes up too
much CPU to get work done, but if I get a spare machine at work and it's not
doing anything, I'll reinstall the client there.
Update: I've read up on some criticism of bitcoin, and while I don't agree with
all of it, some of it does make sense. My main concern with it is that it
doesn't directly represent a physical commodity, rather an artificially created
one (which, arguably, isn't a commodity at all). There's also the problem of
value stored in bitcoin to simply disappear if you were sloppy with backups.
I'm not sure it even can be called money, since it's not a direct store of
value, unlike, say, certificates representing metals, which can be
issued/disappear as the supply of the physical metal increases/decreases.
*** DONE Python project management packages
- State "DONE" [2011-03-21 Mon 02:20]
- State "STARTED" [2011-03-21 Mon 00:58]
To properly package up Python software, I'll need these packages.
python-pip: A Package installer.
- Installed on FreeBSD from devel/py-pip.
python-distribute: A replacement for Setuptools.
- Installed via `pip install distribute.
python-nose: A unit test framework for Python.
- Installed on FreeBSD from devel/py-nose. Next time I'll install via pip
though.
python-virtualenv: Creates isolated Python environments.
- Installed via `pip install virtualenv'.
Added these to fbsd_setup.org. Might want to briefly read up more on these
later (particularly nose). nose is pretty nice, and I'm definitely going to be
using it. However, in work environments, I might not be able to install
packages, in which case, perhaps doctest unit tests might be best
(http://docs.python.org/library/doctest.html).
*** DONE Debian 6 VM
- State "DONE" [2011-03-21 Mon 08:46]
- State "STARTED" [2011-03-17 Thu 08:54]
Considering using this for cloud analytics, as much of that is still a pain on
*BSD, and I'm not even sure if all of it is possible. In either case, I know
some of the development tools I need don't work. This will take a bit of
mindless config work to customize it though.
Setup notes:
- Run the ncurses expert install. Be sure to install the SSH server.
- The rest is pretty standard (basically a subset of my FreeBSD install). It
does require an install of some extra stuff though, like finger, postgres,
etc.
- Find specific versions of packages using `aptitude search <pkg_name>' or
`apt-cache search '^<pkg_name>'.
- To restart the network, run `sudo ifup -a'. If network-manager is being
used, use `sudo /etc/init.d/network-manager restart'.
- Install autocutsel to merge cutbuffer and X selection. Add to .xinitrc.
Setup complete. Debian is a lot more n00b than it was the last time I used it
>10 years ago, but it's still better than n00buntu. Now I just need to get
Hadoop and other cloud stuff installed, but those are in separate tasks. So
far, it seems all this should work.
Update: Had to redo it since my typical 8GB disk is too small to run a cloud
on. Need at least 40GB for the datasets I use.
*** DONE remote host all code repos
- State "DONE" [2011-03-31 Thu 11:50]
- State "STARTED" [2011-03-31 Thu 10:10]
Migrate everything to GitHub. As superior as Darcs is, it looks like it's
lost (I'm declaring GHC's migration to git to be the day this happened).
Maybe I'll still use Darcs for local development or things I don't publish.
I'm not going to bother trying to migrate all my change history from Darcs.
I'll just dump everything in GitHub as is, and go from there.
Tasks:
- Setup git on home Windows box (all my other machines already have it). Add
the git/bin directory to %PATH%. Alternatively just use the Cygwin version.
- If I want a VCS GUI, setup TortoiseGit. I'm not a fan though, since it
doesn't integrate with xplorer2.
- Enter SSH keys into GitHub for all machines and VMs.

- Export my Darcs repo, and start committing to GitHub. I'll want to
reorganize everything into projects for grad_school, practice, and then ones
for everything else.
- Check out the markup syntax for README files.
- Update website repo links.

Decided to just use plain text README files for now.
*** DONE switch to Blekko
- State "DONE" [2011-04-12 Tue 22:51]
- State "STARTED" [2011-04-12 Tue 21:45]
Google results have way too much garbage in them these days. I'll start
searching first at Blekko from now on, which in my experience so far, has much
more useful results.
Conkeror setup notes:
- Added "blekko" and "b" webjump to .conkerorrc.
- Adblock the following URL patterns to get rid of the junk/ads:
http://a.blekko-img.com/*/imgs/promos/*
http://a.blekko-img.com/*/imgs/plugs/*
http://blekko.com/a/yads*
http://www.facebook.com/plugins/like.php*
http://platform.twitter.com/widgets/tweet_button.html*
- Go to prefs and hide stuff I don't want (these are apparently not all stored
server-side, so it's necessary to do it on every machine).
emacs-w3m setup notes:
- It's not possible to modify the defcustoms of w3m-uri-replace-alist and
w3m-search-engine-alist in a w3m-mode-hook function, as these aren't declared
prior to the hook being run.
- Run `cd /usr/local/share/emacs/23.1/site-lisp/w3m; sudo chmod u+w w3m.el
w3m-search.el'.
- Add `("\\`bb:" w3m-search-uri-replace "blekko")' to w3m-uri-replace-alist in
w3m.el,
- Add `("blekko" "http://blekko.com/ws/%s")' to w3m-search-engine-alist in
w3m-search.el.
- Run `sudo rm w3m.elc w3m-search.elc'.
- Run `sudo emacs -Q -batch -f batch-byte-compile w3m.el w3m-search.el'.
- Restart w3m or Emacs
I'll still use Gmail and some of the other Google services, but they've dropped
the ball on search, IMO (too much spam and too many content farms). One thing
Blekko does suck at is having their homepage usable in text mode (hence the
need for the custom w3m setup).
*** DONE ScottradeELITE
- State "DONE" [2011-04-14 Thu 09:32]
- State "STARTED" [2011-04-13 Wed 22:48]
I might use this on days off for day trading and occasionally in the
evening/morning for extended hours trading.
Made a layout. The UI sucks pretty bad though and it's obviously using some
older toolkit. I'll switch to Interactive Brokers (my preferred by far) or
Tradestation after I retire. In the meantime, I might activate the Level II
window, since I do more than 7 trades per month, but only after I'm sure I'll
be trading a lot regularly (I suspect I'll get hit with a fee as soon as I have
a month with a low number of trades).
Notes:
- Use +SYM in a chart window to add a symbol to an existing chart.
- Use (SYM) in a chart window to add a symbol in its own plot to an existing
chart.
- Hold down left click to get the bar information window up on a chart without
having it up all the time (which gets annoying).
- Must disable extended hours in order to chart back more than 10 weeks.
*** CANCELED bsh
- State "CANCELED" [2011-04-30 Sat 21:51]
This may or may not be useful when using JDEE Emacs.
http://gnu-darwin.org/ProgramDocuments/bsh-ug/bsh-ug-content.html
JDEE is too broken for full time use, so don't need this.
*** DONE Pylint
- State "DONE" [2011-05-10 Tue 04:50]
- State "STARTED" [2011-05-10 Tue 05:55]
Using Pylint from the CLI now. Figure out a workflow for resolving Pylint
warnings and getting imports to work properly. Create a custom Pylint
configuration file.
Pylint tutorial: http://www.logilab.org/card/pylint_tutorial
Notes:
- Typically, I'll want to run with `--reports=n --include-ids=y'. The former
usually generates a lot of scroll I don't need and the latter gives me the
IDs of the warning, so I can look it up.
- If a warning isn't obvious, run `pylint --help-msg=<ID>'.
I was going to create a custom config file, but I'm okay with the defaults
(except, perhaps the min length for variables).
*** CANCELED Hadoop Eclipse plugin
- State "CANCELED" [2011-05-14 Sat 04:05]
Supposedly provides some integration for MapReduce/HDFS.
http://wiki.apache.org/hadoop/EclipsePlugIn
Not using Eclipse at work anymore, so skipping this for now.
*** CANCELED Maven 3.0
- State "CANCELED" [2011-05-14 Sat 04:06]
Wait until the new m2eclipse (m3eclipse?) is out to support this.
Not using Maven at work currently, so skipping this. Probably look into it
again at some point in the future though.
*** DONE POSIXLY_CORRECT
- State "DONE" [2011-05-20 Fri 05:38]
- State "STARTED" [2011-05-20 Fri 05:29]
See if I want to define this. I only know about the 512B block sizes used in
df and du, but I'm sure there's other effects. As a FreeBSD-user, I'm not a
POSIX fundamentalist, but I appreciate at least partial compliance. However,
this environmental parameter might flag behavior I don't want.
du/df: No effect for me, since I always use -k or -m explicitly.
tar: Seems to me to make more sense to stick with GNU tar standards.
patch/getopt: Was unable to locate any reference to this in the C source,
though supposedly it affects it.
As I can't really find any significant references to it in the FreeBSD GNU
source besides the stuff I don't care about, I'll skip this.
Made a task for reading the full POSIX standard. Might be worth an attempt.
*** DONE at scheduler
- State "DONE" [2011-06-01 Wed 07:33]
- State "STARTED" [2011-06-02 Thu 05:30]
Starting to use `at' for one-off tasks, particularly for the huge dataset jobs
at work.
Notes:
- Schedule jobs with something like `at midnight' or `at 3AM'. Other date
formats can also be used.
- This pulls up a prompt, which commands can be entered. Be sure to use full
paths or include `cd' commands.
- Conclude a job with C-d.
- Use `atq -V' to view pending jobs.
*** CANCELED pseudo-distributed Hadoop/Hive
- State "CANCELED" [2011-06-02 Thu 10:30]
Set this up on my Debian VM and switch over to it from my pre-built one.
Use this tutorial for Hadoop:
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
Details are also in Hadoop: The Definitive Guide, Appendix A (though it may be
outdated, since I've got an older version of the book).
Use this for Hive:
http://wiki.apache.org/hadoop/Hive/AdminManual/Installation
Update: Canceled, since I don't need a setup of this right now, though it's
possible later. I might still do it anyway if I really have nothing to do some
day and feel like it. In the meantime, I have a ton of more important things
to do.
*** DONE PeerBlock
- State "DONE" [2011-07-20 Wed 20:03]
- State "STARTED" [2011-07-20 Wed 19:59]
Installed this on Windows. Not much to it really and it seems to work okay.
*** CANCELED w3m.el
- State "CANCELED" [2011-08-08 Mon 14:29]
When setting up Emacs on my work Linux VM, I noticed that w3m.el doesn't have
any of the bugs that emacs-w3m does. However, it is missing the
w3m-uri-replace-alist, which I also use. I think no bugs is probably better
than the extra conveniences though.
Update: After using just w3m.el for awhile, I definitely miss the conveniences
of emacs-w3m too much. Since this is my primary web client, I think I'll stick
with it for now, unless more stuff breaks in future Emacs versions.
** Work-related topics
*** DONE Hibernate
- State "DONE" [2011-03-11 Fri 10:37]
- State "STARTED" [2010-11-12 Fri 14:48]
ORM frameworks used to be simple enough to figure out without any dedicated
studying, but as much as I'd like to attempt to get through life without it,
Hibernate is just too all-pervasive in the Java ecology to remain ignorant of
and too feature-bloated to leave to learning by osmosis (at least by my
standards). I am, however, making a directed effort here to only learn the
basics and will revisit the subject again later if I notice too much weakness
in practice.
Tasks:
- Read Wikipedia page on ORM:
http://en.wikipedia.org/wiki/Object-relational_mapping
- Read Wikipedia page on Hibernate:
http://en.wikipedia.org/wiki/Hibernate_(Java)
- Read the official Getting Started Guide:
http://docs.jboss.org/hibernate/core/3.6/quickstart/en-US/html_single/
- Check out the Hibernate tutorial projects linked in the above guide under
the Preface.
- Read the book Harnessing Hibernate.
- Read the book Hibernate Recipes: A Problem-Solution Approach.
- Read some of the PDF version of Hibernate Core Reference Manual and keep it
around for a reference. This is the same documentation installed into
Eclipse and available in the distribution tarball.
- Write at least one decent sample application against a MySQL database. Maybe
use this example, modifying it slightly:
http://www.mkyong.com/hibernate/quick-start-maven-hibernate-mysql-example/
- Find some documentation on Hibernate Eclipse integration or read a few
tutorials on it online.
- Maybe watch some of these videos: http://docs.jboss.org/tools/movies/
Setup Notes:
- Install JBoss Tools Hibernate plugin on Eclipse. See task "Eclipse
customization" for some details on this.
- For a manual Hibernate install (which normally shouldn't be necessary, since
I'll pull it in via Maven), grab a zip or tarball from:
http://sourceforge.net/projects/hibernate/files/hibernate3/
- Don't even bother trying to use the distribution version of Hibernate -- too
many dependencies to manually manage.
Notes (Getting Started Guide and sample projects):
- See ~/src/java/hibernate_tutorial for my version of this code. This includes
modifications to the original master POM to get it to build.
- The tutorial projects don't work out of the box due to JBoss moving their
repos all over the place. Add at least this repo to the POM:
http://repository.jboss.org/nexus/content/groups/public-jboss
- The tutorial also requires specifying the JDK version in a plugin block for
maven-compiler-plugin.
Notes (Quick start Maven + Hibernate + MySQL Example):
- Ran `mvn archetype:generate -DgroupId=com.macroexpand.hbtest
-DartifactId=hbtest -DpackageName=com.macroexpand.hbtest
-Dversion=1.0-SNAPSHOT -DinteractiveMode=false
-DarchetypeGroupId=org.apache.maven.archetypes
-DarchetypeArtifactId=maven-archetype-quickstart' in ~/src/java.
- Created a src/main/resources directory and added hbtest.sql, with code from:
http://www.mkyong.com/mysql/preparing-mysql-data-for-tutorial/ This will
also include the Hibernate XML files.
- Ran `source ~/src/java/hbtest/src/main/resources/hbtest.sql' in MySQL.
- Ran `mvn eclipse:eclipse' in ~/src/java/hbtest to generate an Eclipse
project.
General notes:
- If creating a DAO object that you want to persist, and it has a sequence for
a primary key, the generator tag within the id tag (with a default of
"assigned") can be set to "increment" to not require the primary key to be
defined in code. If you want to actually call the sequence.nextval, then
"sequence" works. There's also "uuid" and several others.
To generate DAO classes:
- Enter Hibernate perspective.
- Create a new configuration with the desired database. For Oracle 10g, add
the Oracle 10g thin client and select the ojdbc14.jar deep within Oracle's
install directory. Set the hibernate.dialect in the main configuration to
match the database type. Test it by expanding all the databases in the
Hibernate Configurations view.
- Go to Run|Hibernate Code Generation. Create a new configuration here. Set
the console configuration to the previously created one. Set the output
directory to <project_dir>/src/main/java. Have a hibernate.reveng.xml file
ready with the tables desired to persist. Select "Reverse engineer from JDBC
Connection", set the package to where the new classes should go, and point to
the reveng.xml file. In Exporters, select "Domain code" and "DAO code".
- Save and run. Move around the files as desired, insert the annotations
necessary, and rewrite the session management code.
I still don't feel super confident using Hibernate. Part of the early
frustration was mostly due to not realizing that the code generator output
almost always needs to be rewritten.  I can see it being easy once you
internalize all the magic keywords, but I'm still not convinced that it
actually saves time overall. I didn't complete all the tasks here, but I don't
think it was necessary mainly because I'm now not going to be using it much
(and hopefully not at all), but also because any further features I need to use
will just be the same painful process of digging through docs and message board
posts. I may pick up some of these tasks later if I ever have to work in it
again.
*** CANCELED JsUnit
- State "CANCELED" [2011-03-21 Mon 02:28]
Not doing JavaScript development anymore, so don't need this.
*** DONE Python
- State "DONE" [2011-04-01 Fri 09:58]
- State "STARTED" [2011-03-14 Mon 13:43]
Already sorta know Python, and have been using it lightly for awhile now. But
since it's now a work language for me, I need to refresh what I already know,
and fill in the basics that I don't.
Tasks:
- Read/do "Learn Python the Hard Way".
- Read some of "Python: The Essential Reference".
- Maybe do the practice problems from:
http://openbookproject.net/thinkcs/python/english2e/
- Use http://docs.python.org/library/index.html (and the other documentation
there) regularly while writing code.
By the end here I should be pretty comfortable with full scale Python project
programming. Python isn't a hard language or anything, so I'm not going to
really do any super-dedicated studying.
I'm not a big fan of Python's OOP model. There's a lot of crufty design here.
The syntax is also a bit awkward. It's nothing terrible, and easy to get used
to, but for being one of the supposedly great new dynamic languages, it's
sloppy.
Decided to skip the 3rd book since the returns on that would be minor. It
looks like some good practice though, so I may do it anyway later.
*** DONE PostgreSQL
- State "DONE" [2011-04-02 Sat 12:00]
- State "STARTED" [2011-03-30 Wed 13:09]
Install this, create a sample database, and maybe write a script or two. I'll
give the manual (available on the project site) a skim. I've used PostgreSQL
before plenty and using it is no more difficult than MySQL, so this is mainly
just to make sure I can set it up and admin the database.
Reading parts of the PostgreSQL 8.4 manual, online at:
http://www.postgresql.org/files/documentation/pdf/8.4/postgresql-8.4.6-US.pdf
Setup notes:
- Installed PostgreSQL 8.4 on Debian. I may also want postgresql-doc-8.4 and
pgadmin3. If I install pgAdmin, I'll definitely want postgresql-contrib-8.4.
- `sudo -u postgres psql' to log into the database. This won't require a
password at fresh install.
- Set a password for the postgres role with:
ALTER ROLE postgres WITH ENCRYPTED PASSWORD 'mypassword';
- To enable some pgAdmin functions, run `sudo -u postgres psql -d postgres <
/usr/share/postgresql/8.4/contrib/adminpack.sql'.
- To enable remote access, edit /etc/postgresql/8.4/main/postgresql.conf with
listen_addresses = '*', then edit /etc/postgresql/8.4/main/pg_hba.conf,
adding the line `host all all <IP_ADDRESS>/24 md5' (this is for the 192.168.1
subnet -- change to whatever makes sense).
- Address ident authentication issues by editing
/etc/postgresql/8.4/main/pg_ident.conf. This maps system users to DB users.
Or, more easily, just edit pg_hba.conf, comment out ident lines, and add
`local all postgres trust' and maybe `host all all <IP_ADDRESS>/32 trust'.
- Restart PostgreSQL with `/etc/init.d/postgresql restart'.
- Now I should be able to connect with pgAdmin and/or create databases and
login with psql as my normal user.
- For example, create a database with `createdb -U postgres bmdb', then log
into it with `psql bmdb postgres'.
- If I want to be able to run PostgreSQL commands and log into psql without
specifying a user, I can just add a PostgreSQL user of the same name as my OS
user. Alternatively, I could set $PGUSER to whichever login I want.
Notes:
- The built-in point type is a coordinate pair, which could be useful for
geo-data.
- Insta-load data from text files using COPY, e.g. `COPY weather FROM
'/path/to/somefile.txt';'.
- Use INHERITS clauses in CREATE TABLE statements to create a table that has
all of the parent's columns, plus some additional ones.
pgAdmin sucks pretty bad, but it gets the job done most of the time. I'll
probably try to stick to the command line as usual.
Of the PostgreSQL manual, read the sections I and II. A painfully boring read,
but this manual will be good to keep around for reference.
*** CANCELED Spring Framework
- State "CANCELED" [2011-04-05 Tue 09:02]
- State "STARTED" [2011-01-23 Sun 22:53]
Ugh, Spring... Hopefully I can position myself in my remaining work years to
minimize or exclude this from my day.
Read:
- The Wikipedia page on Spring: http://en.wikipedia.org/wiki/Spring_framework
- This introduction:
http://www.theserverside.com/news/1364527/Introduction-to-the-Spring-Framework
- The majority of Spring Framework 2.5 Reference Manual:
http://static.springsource.org/spring/docs/2.5.x/reference/index.html
- The book Spring Recipes: A Problem-Solution Approach (2nd Ed.).
- The What's New chapter of the Spring Framework 3.0 Reference Manual:
http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/html/
- Consider reading Spring Enterprise Recipes if I feel still a bit lacking
after the above.
PDFs of the Reference Manuals are also available at:
http://www.springsource.org/documentation
I seem to have managed to sufficiently alter my work such that Spring is no
longer necessary, at least for now. I did do some of the sub-tasks here, but
it's unlikely I'll retain anything in a few months, so I'm calling this
canceled. It's entirely possible that Spring could force its ugly presence
upon me again in the future, at which point I'll reopen this.
*** CANCELED HBase
- State "CANCELED" [2011-04-19 Tue 16:26]
Not using. Might read up on it anyway later, as it might be a solution for
some of the low through-put/low latency applications -- unlike Hive which is
for the reverse.
*** DONE Apache Hive
- State "DONE" [2011-04-29 Fri 05:35]
- State "STARTED" [2011-04-21 Thu 07:53]
Hive is a "data warehouse infrastructure built on top of Hadoop". Data is
stored in HDFS (with all the benefits that entails) and ad-hoc querying is
provided using HiveQL (an SQL-like language). Queries are translated into
MapReduce jobs. These can be slow (depending on the cloud and data), thus it's
not suitable for real-time querying, nor does Hive provide the majority of
RDBMS features (even basic ones, like foreign keys). However, it is a useful
method for storing and interfacing with data on Hadoop, as it abstracts a lot
of the lower-end programming necessary when writing MapReduce code and
simplifies data management.
Tasks:
- Grab the source and read the README.txt.
- Read everything on the Hive wiki (starting at
http://wiki.apache.org/hadoop/Hive/GettingStarted).
- Find some extra info on UDFs/UDAFs.
- Watch the Hive videos linked on the Getting Started page.
- Read all of the example queries to make sure I didn't miss any important
syntax.
- Read the entire Hive commits, user, and dev mailing lists archives
(pre-Hadoop sub project). Only reading stuff that is relevant to Hive
architecture or HiveQL.
- Might read some of the source code.
Setup:
- Either run `git clone https://github.com/apache/hive.git' or `svn co
http://svn.apache.org/repos/asf/hive/trunk hive' (at least, for trunk).
- `cd hive_trunk', then `ant clean package'. This takes forever, since ivy has
to grab a ton of dependencies.
- build/dist in now ready to deploy somewhere.
- Instead of the above, just grab the binary distribution and upack it
somewhere.
- Set $HIVE_HOME to wherever I deploy Hive and add $HIVE_HOME/bin to $PATH.
- Make sure Hadoop is already properly installed and working (and that
$HADOOP_HOME points to the right place with $HADOOP_HOME/bin in $PATH).
- Run these commands to create the expected Hive HDFS directory structure:
- hadoop fs -mkdir /tmp
- hadoop fs -mkdir /user/hive/warehouse
- hadoop fs -chmod g+w /tmp
- hadoop fs -chmod g+w /user/hive/warehouse
- Edit $HIVE_HOME/conf/hive-site.xml (compare these to the defaults in
conf/hive-defaults.xml).
- Add any user-specific Hive startup options in $HIVE_OPTS (added "-hiveconf
hive.root.logger=ERROR,console" for my setup).
- I had problems compiling Hive from source, and just grabbed the binary
distribution. However, my problems were network-related, so I may retry
later.
- Modified .emacs to include:
;; Add an auto-mode for the typical HiveQL extension.
(add-to-list 'auto-mode-alist '("\\.hql$" . sql-mode))
General notes:
- HiveQL-specific notes in test_ddl.hql, test_dml.hql, test_q.hql, and
test_full.hql.
- See the two sets of CLI flags with `hive -h' and `hive --help'.
- Use SET to set various Hadoop configuration variables. Use `SET -v;' to list
all variables/settings. Just `SET;' will list the non-default variables.
- For processing a job locally, which can be quicker for a very small dataset,
use `SET mapred.job.tracker=local;' (only works in >0.7). This uses space in
mapred.local.dir. It's also possible to enable this to automatically use
local mode for small data (<128MB, <4 map tasks, <2 reduce tasks by default)
with `SET hive.exec.mode.local.auto=true;'. These are disabled by default.
- Logging: Can configure the logger for the session with hive.root.logger,
e.g. `-hiveconf hive.root.logger=INFO,console'. This is an exception to the
ability to set a variable with SET (since it's initialized at start). Errors
by default go to /tmp/<user>/hive.log (on the local FS).
- Java regexes work on some commands like `show tables '<regex_string>''. See:
http://download.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html
- Due to the use of Derby, only one connection can access Hive (either through
the CLI or through HQL scripts). A connection creates lock files in
$HIVE_HOME/metastore_db. One way around this currently is
HiveDerbyServerMode.
- Lots of HiveQL examples in build/dist/examples/queries and more in the source
at ql/src/test/queries/positive.
- `hive -f' doesn't like dangling comment blocks (comment blocks without any
code under it).
- If using Hive to run Python scripts, use sys.path.append(os.getcwd()).
- In the USING..AS clauses (where you can put a Python script), pretty much
anything that writes to stdout can be here (e.g. /bin/cat).
- Hive now supports WHERE..IN. Prior to that, it required `find_in_set(col1,
'abc,xyz') <> 0', with coalesce() if the column has NULL values.
- Full list of Hive types: http://wiki.apache.org/hadoop/Hive/HiveQL/Types
- To use external variables, e.g. `describe ${hiveconf:var}' in a file called
t.hql, run the script with `hive -hiveconf var=mytable -f t.hql.
- Date types aren't yet supported. Must use strings instead for now.
- See the EXPLAIN plan for any query with `EXPLAIN [EXTENDED] <query>'.
- The name node will be in safe mode 30 seconds after startup. Turn this off
manually with `hadoop dfsadmin -safemode leave' or just wait.
- Use `insert overwrite local 'local_directory' select ...' to dump stuff to
local files. This output will have control characters in it, so I usually
insert commas in the select and then filter out the \x01 chars with a Python
script that does a `re.sub(r"[^a-z0-9-';.,]", "", line)'.
UDFs (user defined functions):
- Get a description of functions with `describe function <function_name>;'.
`describe function extended <function_name>' for full details.
- There's some weird rules with HiveQL UDF/UDAF use. Keep this reference for
work-arounds: http://wiki.apache.org/hadoop/Hive/LanguageManual/UDF
- Start Hive with `--auxpath /path/to/file.jar' to add jars to the classpath.
An alternative is to define $HIVE_AUX_JARS_PATH (comma delimited). This is
useful for packaging UDFs for use within Hive. If doing this, it's then
necessary to do `create temporary function <func_name> as
'com.company.pkg_name.ClassName';'.
- To call a UDF by itself, do something like `select to_date('1900-01-01') from
<some_small_table> limit 1;'. Ideally, some_small_table would just have one
row and it might make sense to create a dummy table.
- UDFs can be written in Java or Python (and other languages).
- A UDAF is an aggregate function, for which there are special considerations
for (due to MapReduce). A UDTF is a table-generating function.
I'm now able to be productive with Hive. Read some of the mailing list but it
seemed like a non-optimal use of time, as was reading the source. The rest of
the expertise will come with using it all the time. I'll still read the Hive
wiki and mailing list occasionally when I have some free time though.
*** DONE Apache Hadoop
- State "DONE" [2011-05-06 Fri 06:22]
- State "STARTED" [2011-03-21 Mon 02:34]
Need to know this in great detail. I'll call this done when I'm familiar with
most of the common settings, know how to load stuff into HDFS, have
internalized the HDFS basic commands, understand Hadoop internals to some
extent, and have a list of online resources to consult when I need detailed
info.
Hadoop involves two components: HDFS and MapReduce. My approach is:
- Survey the Hadoop ecology, learning the problem space it addresses and how it
fits in with other cloud computing technologies.
- Read up in detail on HDFS.
- Install Hadoop on a virtual cluster.
- Read up in detail on MapReduce.
- Find the Java APIs to program against Hadoop.
- If I have time, write some practice code.
Tasks:
- Attended a work class on this topic. This was pretty good for an overview.
- Reread the class slides later.
- Watched a few videos online about various Hadoop stuff and Google's
MapReduce. There's a pretty good Google Code video series called "Cluster
Computing and MapReduce".
- Read the Hadoop wiki overview.
- Read this Hadoop Tutorial: http://developer.yahoo.com/hadoop/tutorial/
- Read the book "Hadoop: The Definitive Guide".
- Read the MapReduce tutorial:
http://hadoop.apache.org/common/docs/r0.18.3/mapred_tutorial.html
It turns out I'm not currently directly writing any Java code against
MapReduce/HDFS (only Hive and Hadoop streaming in Python), so I'll skip
mastering that for now and come back later and finish it. I'll also defer
setting up my own psuedo-distributed cluster (at work, admin is handled
externally).
*** DONE Weka
- State "DONE" [2011-05-09 Mon 08:52]
- State "STARTED" [2011-05-06 Fri 07:17]
Weka is a suite of machine learning algorithms. I've used this back in grad
school, but that was a long time ago.
http://www.cs.waikato.ac.nz/ml/weka/index_documentation.html
Tasks:
- Read Wikipedia page on Weka:
http://en.wikipedia.org/wiki/Weka_(machine_learning)
- Read Weka Manual 3.7.3.
- Review all of the algorithms used by Weka.
Setup:
- Just installed binary version via apt-get on GNU/Linux.
- Ensure that weka.jar is in $CLASSPATH.
Notes:
- See book notes under Weka Manual entry.
- Generally, it's much easier to just use the Weka GUI to select parameters and
such, but running `java weka.<package.class>' is pretty good about showing
them too. Flag input files with -t.
- I see many possibilities using Weka for writing trading bots.
- Kappa statistic (Cohen's kappa coefficient): A statistical measure of
inter-rater agreement. 1.0 signifies complete agreement.
- A history of algorithms and flags used gets written to weka.log.
Couldn't find a list of all of Weka's algorithms with descriptions. I might
make one of these myself later, but that's a lot more work.
*** CANCELED jQuery
- State "CANCELED" [2011-05-10 Tue 05:57]
Deployed form in ~/public_html/inc/jquery-1.4.2.min.js.
While jQuery would be useful to know in general, I don't write JavaScript at
work anymore, and thus can put this off until later (by which time, there may
be something better anyway).
*** DONE Afghanistan war
- State "DONE" [2011-05-24 Tue 06:16]
- State "STARTED" [2011-05-20 Fri 07:57]
I already know a lot about this, but now that it's my job, I'll add some more
focus on it to fill in the gaps. Skipping the Soviet-Afghan war background as
I already know that in great detail.
Tasks:
- Update on unclass recent war history. Using The Institute for the Study of
War papers from the last year, mainly.
- Read: http://en.wikipedia.org/wiki/Pre-Islamic_period_of_Afghanistan
- Read: http://en.wikipedia.org/wiki/Islamic_conquest_of_Afghanistan
- Read up on differences between Shia and Sunni Islam. Some info here:
http://en.wikipedia.org/wiki/Shi'a–Sunni_relations
- Read:
http://en.wikipedia.org/wiki/Civil_war_in_Afghanistan_%281996%E2%80%932001%29
- Read up on the Taliban and various factions.
- Read up on the various other groups: IMU, LeT, etc.
- Afghanistan geography. Maybe grab a provincial map and keep filling it in
until I can do it by memory. At least become familiar with FATA provinces.
http://www.globalsecurity.org/military/world/afghanistan/images/afghan-provinces-map.gif
Notes:
- Historic Bactria, supposed birthplace of Zoroaster, was the origin of the
Kushan empire and now the modern day Balkh province.
- The main differences between Shia and Sunni: Shiites believe only the 4th
caliph, Ali, and his heirs are the rightful successors of Mohammad, whereas
Sunnis believe Abu Bakar (and 3 others, including Ali) held that position.
Also, Shiites believe the Mahdi is here and in hiding, whereas Sunnis believe
he hasn't come yet. In addition to these, there are numerous small cultural
differences and which hadiths (narratives) are accepted.
- The Taliban are anti-Shia and primarily Pashtun.
- The primary Taliban division is between TTP (Tehrik-i-Taliban Pakistan) and
Afghan Taliban. The primary difference is the theater (FATA
vs. Afghanistan). Both factions are enemies of Iran (for the most part), and
TTP is currently in open conflict with the Pakistani regular army.
- IMU seems like not a significant entity on its own, though its tight ties
with the Taliban bolster its fighting strength a little. LeT is primarily
anti-India, and not a huge player in the Afghanistan war.
Decided to skip the province memorizing for now. After all this reading, I
know where most of the important ones are and recognize their names, which
should be enough. I'll try to read at least a paper about the war every few
weeks to keep up on the latest.
*** DONE Apache ZooKeeper
- State "DONE" [2011-05-29 Sun 10:24]
- State "STARTED" [2011-04-21 Thu 07:37]
Just need to know about what it does, not the specifics on how to setup/admin
it (though knowing that as well might be nice).
ZooKeeper is a service and library for building distributed data structures. A
number of idioms exist for this, and ZooKeeper implements these for you. After
reading the chapter on it in Hadoop: The Definitive Guide, reading the intro
pages on the ZooKeeper wiki, and reading some Cloudera blog posts, I still am
not completely sure how it integrates with Hadoop, or even if it does, but
since I'm not using it at work, I'm capping my effort here.
*** DONE read all RHEL info docs
- State "DONE" [2011-06-13 Mon 10:06]
- State "STARTED" [2011-05-10 Tue 06:17]
Read every info page on RHEL. Doing this between Hive queries at work. Skip
the C library info pages. The big ones are the Emacs Manual, which I already
read, and the two elisp manuals (though apparently the tutorial is missing on
that version of RHEL), which I broke into separate tasks. This isn't really a
work-related task, but I wouldn't be doing it if not sequestered in a SCIF all
day.
Read the info page for every GNU utility that has one, provided it was
something I didn't know completely.
*** DONE read all RHEL manpages
- State "DONE" [2011-07-08 Fri 09:39]
- State "STARTED" [2011-06-13 Mon 10:41]
Might as well, since it's mildly productive and passes the time.
Notes (incomplete due to not having this file on hand):
- anacron is similar to cron but is intended for machines that are not always
on.
Done for now. Read a ton of these, but didn't learn much. May reopen this
later if I find myself stuck on an RHEL box all day.
*** DONE Udig
- State "DONE" [2011-07-26 Tue 14:33]
- State "STARTED" [2011-07-25 Mon 15:11]
Installed on Windows. Doing the built-in tutorial, which includes some sample
data.
Finished the Quick Start tutorial. I can probably just lazy-load the rest.
** Games to play
*** DONE Light Crusader
- State "DONE" [2011-02-27 Sun 20:10]
- State "STARTED" [2011-02-27 Sun 19:47]
Jumping puzzles in an RPG = me quit.
*** DONE Sudoku
- State "DONE" [2011-03-30 Wed 00:19]
- State "STARTED" [2011-03-18 Fri 11:18]
Never played sudoku before, so giving it a try. Installed the console version
on FreeBSD.
An okay time-waster. Not much thinking involved though. It's possible for a
"fiendish" difficulty game to have a valid solution given the set cells, but
that solution not be the one the game determines to be a correct answer. I
consider this mode bugged as a result.
*** DONE Fallout: New Vegas
- State "DONE" [2011-04-09 Sat 20:48]
- State "STARTED" [2010-12-17 Sun 20:34]
IMO, better than Fallout 3 overall and a great game, but still has some
problems. This game captures the original Fallout feel better than 3 (not that
I'm particularly hung up on that), has vastly improved weapons, and is just a
more interesting world to interact with.
Complaints:
- The true-to-the-series, massive collection of bugs everywhere. Even with the
patch, any play-through encounters a ton of game freezes, quest system bugs,
stuck NPCs, etc, forcing reloads and restarts.
- Numerous graphical issues like with draw distance and model clipping.
- Not many of the NPCs really had all that much character. The previous
Fallout games weren't that great at this either, but 3 did it slightly
better.
- Many areas have almost nothing to do, as if they were just slapped together.
- Less game real estate than 3, which itself was a bit lacking for my tastes.
Particularly in the western desert setting, stuff should have been way more
spread out. Both Bethesda games squish real life geography all close
together. Boulder, CO is over 700 miles from Las Vegas, NV, for example, not
the approximate 1 mile it seems to be in the game.
- Other annoying geographical errors. The desert around Las Vegas is very
flat, not surrounded by mountains.
 This is just lazy,
particularly since they already had all the previous game's models to use.
All they'd have had to do is map edit them up.
- No really decent housing (that I found so far). I ended up moving into the
gas station at Goodsprings since it was far superior of a location than the
other bases I got. The Lucky 38's upgrade menu was also bugged and unusable.
- Only 3 casino games (+1 custom game)? At least craps should have been
added. The way it is now, casinos get old pretty fast. The games that were
included were done well, however.
- It's not humanly possible to reload a single action revolver that fast (same
goes for a lot of the other guns).
- The ending is kinda bleh.
May play again some day in power-game mode.
*** DONE Syndicate Wars
- State "DONE" [2011-05-14 Sat 03:28]
- State "STARTED" [2011-04-16 Sat 22:21]
Played this a lot back in the mid-90s, but never beat it thanks to the
timebombs on the orbital station. I'll also try the Church campaign.
Setup:
- Grab the Syndicate Wars DOSBox config from:
http://syndicate.lubie.org/downloads/swars/swars_dosbox_6_5_conf.zip
- Install DOSBox if it isn't already, and create a shortcut to DOSBox.exe. Put
the .conf file in the DOSBox install directory. Add `-conf
dosboxSyndWars.conf' to the target line.
- Create an install dir and start DOSBox using the shortcut and run `mount c
c:\bin\games\swars\'.
- Put the install CD in and run `mount d e:\ -t cdrom -usecd 0'.
- Go to the D: drive and run install.
- Set both sound and music drivers to SB16 and configure automatically. The
tests should work.
- Do a full install.
- Edit the play.bat file to include the line `mount d e:\ -t cdrom -usecd 0'
first.
- Edit the shortcut target to include `c:\bin\games\swars\swars\play.bat'.
- Before playing, at least change the resolution to 640x480, activate deep
radar, and remap the camera to WASD, zoom to TG, and group agents to Q.
- Set cycles to around 70000.
- Tried the native port as well, but it runs too fast.
- Can't beat the last 3 missions (you have to do them all in a row without
losing anyone). I'm sure it's doable, but I don't feel like doing it over
and over. Backing up the saved game, and might try again one day when I
don't have a job.
Great game, even now. Supposedly a sequel is finally in the works, so will
keep an eye out for that.
*** DONE Vampire the Masquerade: Redemption
- State "DONE" [2011-06-03 Fri 23:26]
- State "STARTED" [2011-05-29 Sun 14:09]
Played this 10 years ago, but ran through it quickly, using cheats, and getting
the bad ending. Doing it properly this time, with some treasure chest scumming
and powergaming of stats/skills/gear.
Not a great game, but still a good one even now, despite a lot of bugs and
problems. My power-gaming run made completing this extremely easy. Probably
could've done it solo.
*** DONE Vampire the Masquerade: Bloodlines
- State "DONE" [2011-06-10 Fri 23:42]
- State "STARTED" [2011-06-04 Sat 20:53]
I recall not liking this game back in 2004, but will give it another try.
Update: Now I remember why.  Quitting.
*** DONE Stars! 2.7
- State "DONE" [2011-07-10 Sun 03:28]
- State "STARTED" [2011-07-03 Sun 18:36]
Used to play this 4x space game 10+ years ago quite a bit.
Note that the serials I had from before don't work. First ensure the directory
isn't read only, then use "E4TP99BW". If it's still being ghey, delete the
GlobalSettings value in the STARS.INI file created in the C:\Windows directory,
then restart and re-enter the serial. This is also a 16-bit game, so it
doesn't run on 64-bit versions of Windows. Playing it on an XP VM. There's an
active channel for the game on Freenode that was helpful for a few things too.
Played a few scenarios. Definitely a great game and I'll keep it around to
play occasionally. There's a couple things I'd change, but overall it's just
as good or better than most modern 4x games. My main complaints are just the
amount of mouse clicking involved despite some advanced automation and the fact
that once the screen gets really full of stuff, it's hard to tell what's going
on without a lot of clicking or digging through report screens.
*** DONE Deja Vu
- State "DONE" [2011-09-24 Sat 11:12]
- State "STARTED" [2011-09-21 Wed 14:02]
Playing the NES port. Already beat it long ago, but it's been like 20 years.
Crappy game. Not sure why I liked it as a kid.
*** CANCELED Jones in the Fast Lane
- State "CANCELED" [2011-10-11 Tue 11:13]
Tried the Flash version of it to see if I liked the game style. This could've
been interesting, but I'd rather it just be a life sim and not a competition
against another player or the computer. The actual game content itself is
pretty bleh too.
*** DONE Defense of the Ancients
- State "DONE" [2011-12-07 Wed 10:39]
- State "STARTED" [2011-11-07 Mon 13:22]
OK creep/tower mod. Doesn't have the long-term appeal that straight tower
defense does though, at least for me. The biggest problem is the computer
controlled characters.
** General life goals
*** DONE rebuild firearms collection
- State "DONE" [2011-02-13 Sun 13:43]
- State "STARTED" [2010-11-13 Sat 12:25]
I'm not going to build a full collection and will only own 3 handguns, each
which serves a distinct purpose:
- Sig Sauer p228: Shoots a cheap, reasonably-effective anti-personnel round. I
can effectively engage multiple targets at ranges up to 50+ yards with this.
I'm only firing this just enough to remain competent with it, as I find
semi-autos predictably boring to use (especially at ranges). Accessories: 3
magazines, an OWB holster, 500rds FMJ, and 50rds JHP.
- Ruger New Model Single Six with 4 5/8" barrel: Target and utility gun.
 Shoots just about any .22
ammo available. Post-retirement, this will be handy for shooting squirrels,
rabbits, and other animals that eat food intended for me (though a rifle
would be better for this). The extra .22 magnum cylinder enables it to take
out medium-sized targets if necessary. Being a .22, it's main purpose is for
practicing cheap. I'll be putting many thousands of rounds through this
thing, which is why this model was selected and why it was bought new.
Accessories: western-style holster with belt, 5000rds LRN, 100rds HP, 50rds
shotshell, and 50rds JHP .22 magnum.
- Colt Detective Special: Concealed carry revolver. This one was selected
since the Colt D-frame is roughly between the S&W J- and K-frame, and holds 6
rounds instead of the J-frame's 5. Colt DA revolvers also tend to increase
in value faster than inflation, which is a nice plus, though I intend to keep
mine forever. Accessories: IWB holster, 300rds FMJ/LRN, and 50rds JHP.
Finished buying all of these and most of the accessories. No rifle means no
long range capability, and I'll add one when/if I get a retirement property. I
wouldn't want to get something now that I can't use later. I got most of the
accessories, but not all of the ammo, which I'll get later since that much
would weigh a lot.
Update 2012-05-20: Sold the SIG P228 for $500 to a guy I met in Afghanistan. I
may replace that with a 1911, but I don't really feel unarmed with just the
revolver. I noticed I don't like firearms as much as I used to when a kid, so
maybe this is fine.
Update 2012-09-24: Bought a Colt Series 80 Government Model. It's pretty nice.
Update 2012-11-05: Though I wasn't in the market for it, I found a good deal on
an FN SCAR 16s from a desperate seller. It's definitely a nice rifle.
*** DONE $3/day food
- State "DONE" [2011-02-28 Mon 15:15]
- State "STARTED" [2011-01-05 Wed 10:02]
Apparently it's possible to even do $1/day food, but that'd be tough in my area
with food prices so insanely high and with no ability to grow any significant
quantity of food myself. I've already been trying to reduce these costs and am
currently at $5.28/day. If I can manage this, it'll save over $800/yr.
Some methods:
- Eat as much food at work as possible, except candies and donuts (since that
conflicts with my other goals).
- Buy tons of tomato soup cans whenever they're under $1/each (preferably
between 50 and 75 cents).
- Buy large bags of potatoes of size 10lbs or more. 10lbs should cost $5 at
most. Buy large bags of rice of at least 25lbs.
- Make salads from various combinations of: cucumbers, green peppers, lettuce,
carrots, radishes, and onions. Use vinegar, oil, black pepper, and oregano
instead of normal salad dressing. I can skip the lettuce to save some money
though, since it doesn't really provide anything I need to live off of.
- Make sandwiches from $1/loaf bread and tomato, jalepeños, and mustard. Get
some plastic container to bring these to work.
- Buy the cheapest tortillas and make vegetable burritos out of them. Cans of
refried beans that are under $1 are also good in combination with these.
- Buy the cheapest, largest box of spaghetti. Make sauce with tomato paste,
water, and various spices.
- Whenever I have some random leftover vegetables, combine them with tofu and
make some stir-fry. Never eat more than half the block myself though (save
the rest if there's no one else available to eat it).
- At least eat some non-grain products every day. Saving all this money won't
pay off if I end up all sickly and malnourished. However, pre-modern day
humans attained the majority of their sustenance from grain (several times
the percentage of today), and I'm okay with doing the same.
Like other stuff here, this change is permanent (unless I find myself
overwhelmed with piles of cash at some point in the distant future). If I ever
want a nice meal or to eat out, I'll need to reduce the money expended on other
days to compensate for it.
Successfully implemented this, at least for food. Miscellaneous supplies like
bathroom products are still adding a bit to my expenses though. However, I've
discovered I can purchase most of these at the dollar store and reduce their
cost to around $10-15/mo. I should probably just do a full monthly budget
instead of attacking each expense by itself though.
*** DONE balcony garden
- State "DONE" [2011-03-21 Mon 03:03]
- State "STARTED" [2010-12-24 Fri 11:15]
As much as I hate doing this, it's a smart thing to do for these reasons:
- Food prices keep going up. I've seen up to a 33% rise in some items I
regularly buy just in the last 3 years. For the really expensive things, I'm
just going to stop eating them. Even many fresh produce prices are painful
now. Just raising a few plants in buckets, if successful, will easily put
another hundred dollars or more right in my pocket every year.
- Having a garden that takes care of a good portion of my food needs is an
integral part of my retirement plan. I need to start developing those skills
as soon as possible.
- A lot of fresh produce is waxed and not available otherwise.
- I know from eating relatives' garden vegetables that they taste considerably
better, as they weren't picked unripe.
- More money saved = earlier retirement.
The plan is to spend as little as possible. So far, I only bought one garden
trowel for $3 and one pack of tomato seeds off Amazon for $1.60. The
containers are spare cat litter buckets and drink containers cut to size. The
potting soil I'm stealing from around the neighborhood. The rest of the seeds
are from vegetables I was eating anyway.
Potentially good balcony plants to try: garlic, tomatoes, green peppers,
jalepeño peppers, cayenne peppers, radishes, scallions.
Results: Cat grass is easy to grow, but doesn't really save any money. It does
keep cats from eating my other plants though. In the winter I planted 1
garlic, 2 tomatoes, 2 green peppers, and 5 radishes. Most of these died due to
cat eating. Only the garlic worked out okay. It turns out I don't have enough
sunlight to grow most plants, especially indoors. I'll stick to garlic and
radishes until I move. Thus, I'm not really saving much money, but I did learn
a few things. However, I can produce some food and it'll save a few
dollars/month which is better than nothing.
** Macro-goals
*** DONE attain competence in new job skills
- State "DONE" [2011-03-14 Mon 10:28]
- State "STARTED" [2010-03-12 Fri 10:17]
All other major tasks take a backseat to learning everything needed to perform
work with minimal effort when starting a new job. The goal here isn't to
attain complete mastery, as I can pick that up as I go along without much
effort. This entry is to keep track of overall progress towards this goal --
see individual topic entries for details.
Topics:
- base JavaScript language
- Tomcat
- Ext JS
- base Java language refresh
- survey intelligence analysis
- ActiveMQ
- Maven
- super-brief Nexus survey
- REST services
- Hibernate
- statistics
- statistical learning models
- Spring
- R
- JUnit
Optional topics:
- Jackson
- Hudson
- Mockito
Almost finished this, but since I'm moving to a different project, I'm going to
start a new macro-goal with all of those things. I do still want to learn R
and statistics in particular a bit better, but that can wait. For about a year
of self-directed learning, this went pretty good.
*** DONE attain competence in new project skills
- State "DONE" [2011-07-25 Mon 08:47]
- State "STARTED" [2011-03-30 Wed 10:17]
Moving to a DARPA project, so I'll need to learn an mostly entirely different
set of work-related technologies. This could last up to two years, so even if
I spend half of that time getting really good with this stuff, it will be time
well spent. Most of these things are cloud analytics-related.
Topics:
- Python
- PostgreSQL
- Apache Hadoop (HDFS and MapReduce)
- Apache Hive
- Weka
- HBase
- Tableau
- Apache ZooKeeper
- Mahout
- RapidMiner
Ancillary tasks:
- Setup a work Debian VM.
- NSA CloudBase
- Finish learning R.
- Finish learning statistical learning models.
- Finish learning statistics.
After this, there's a list of related technologies that aren't actively used at
work, but would be useful to at least know in outline (for technology selection
purposes and larger context). These are listed in the Work-related Topics
section.
Still need to do more math self-learning and there plenty of ancillary stuff to
learn, but I've got all the main topics down now.
*** DONE year-end review for 2011
- State "DONE" [2012-01-02 Mon 04:54]
- State "STARTED" [2012-01-02 Mon 04:07]
Observations for 2011:
- Work life was divided into working on RTRG for a few months, changing jobs to
DARPA, then deploying to Afghanistan. Once the last phase began, work and
normal life merged.
- Net worth more than doubled (multiplied by 2.23). I'm well ahead of schedule
on getting my target amount for retirement (by about 1.5 years). More
importantly, I've secured a position that reimburses me at a rate where I'll
hit that amount around mid 2012.
- Successfully made a ton of money in an overall flat market for the year.
- Further optimized possessions for an estimated net reduction of around 20%
(by weight).
- Optimized several aspects of consumption, which is pretty close to minimal
now. There's are a few things to fix here remaining, but none that I can act
on in the current situation since I live out of a suitcase in random places.
- Mastered Python and further developed several other work-related knowledge
bases: cloud computing, statistics, algorithms, and functional programming.
- Developed a passable public speaking ability. Though I'm still no master at
this, I can now deliver a speech or brief to a full room. As I suspected,
this was just a matter of practicing it.
- Learned how to survive (so far) in Afghanistan and among the military.
- Overall, this year was about maximizing net worth, which was the intended
primary goal. I did manage to develop some permanently useful skills on top
of that (and some that will be useless later). In my opinion, this vastly
tops the productivity of last year and all major decisions made were the
right ones.
Major goals for next year:
- Finish acquiring retirement target amount.
- Take a huge amount of time off once I get back from Afghanistan. By then,
I'll have gone 1.5 years without a single day off, and will need that time to
flush some garbage out of the brain. All warzone-specific knowledge can be
safely deprecated at that time. Also use this time to rebuild normal life.
- Do whatever is necessary to be able to stay in Afghanistan until the end of
November, 2012 to save money on taxes. If this doesn't work out, it won't be
a disaster, but I do absolutely need to stay in country until August. Suffer
whatever pain comes in the meantime (and there's sure to be a lot of it).
- Maybe come up with a detailed exit plan from work. Currently, this involves
going back to DARPA for awhile. During that time, I'll come up with the
specifics. If I do that, this will get pushed off to next year, which is
fine too, as it's better to get the plan right rather than sooner.
- Do taxes remotely.
- Do the best I can with learning new stuff. If everything goes according to
plan, I'm somewhat writing off 2012 from learning much of anything. If I can
maintain my existing knowledge/skills, then I'll be happy with it. However,
I do occasionally find myself with free time, so I'll be making as best use
of that as I can. Once I get back to the US, the master plan will go back
into effect. I've found it's tough to develop Lisp without internet
connectivity, so I'm switching back to Haskell for the time being.
- Keep total expenses for the year under $4k (slightly more if I have to pay
bribes for VISA renewals).
- Don't let being in Afghanistan cause any permanent brain damage.
- Don't die.
* Project List (2010)
** Books: Computer Science/Programming
*** DONE The Art of Unix Programming
- State "DONE" [2010-02-05 Fri 00:26]
- State "STARTED" [2010-01-29 Fri 14:55]
Once I realized this book wasn't going anywhere and was mostly filler, I
started skimming it pretty quickly. There's a few okay parts later on, but
even these are shallow introductions to their respective topics.
Wrote the following for this book's amazon.com review:
As someone who's programmed on Unix for many years, I've known about esr for
some time, and probably should thank him for being part of a chorus that
encouraged me to learn things like Lisp. That said, my honest opinion of this
book is that it's a waste of paper and approximately the quality of a typical
esr blog post.
First of all, this book is not really about Unix programming. A full half of it
is dedicated to rehashing the Unix philosophy, history, and community. Yes,
these topics should come up, but at some point I want to read about Unix
programming, which is why I am reading a book with that phrase in the
title. None of this stuff has anything to do with programming, and is just as
applicable to end users.
The other half is supposedly programming related, but upon closer inspection,
mostly isn't. Major topics include things like tools, config file formats, and
a lot of general stuff that applies in non-Unix development too (like VCS and
network protocols). Where's the system programming tips, POSIX standards,
standard libraries, ioctl, BSD sockets, security considerations, and any number
of hundreds of other things anyone who's programmed on a Unix-variant
encounters? Yes, the book claims to not want to talk about these things, but
guess what, that's what Unix programming is all about.
Languages (you know, the thing you actually program in) are completely glossed
over, except to just list a few common ones. Half of the ones mentioned are so
portable, that programmers in them can ignore OS peculiarities the vast
majority of the time (Java, Python, Emacs Lisp). C, C++, and shell are very
tightly coupled to Unix, and much could have been said here, but isn't. Instead
of code, we're treated to reams of config file examples and other filler.
The various aspects of The Unix Way could have been stated in a single
chapter. Someone interested in becoming a Unix programmer needs to know where
to go to find documentation, what development resources he has available, and
other practical things. An MS programmer reading this book would be just as
clueless on how to start programming on Unix as before. Go pick up Advanced
Programming in the UNIX Environment if you want to learn this subject.
*** DONE Essential C++
- State "DONE" [2010-02-26 Fri 08:25]
- State "STARTED" [2010-02-23 Tue 10:44]
Skipped the first few chapters of stuff I already know. I didn't like this
book so much at first, but it grew on me some. It's not bad for a quick
refresher, but I'd rather the examples be more self-contained and more varied.
Of course, it's not a super-excellent programming book, but compared to most of
the trash in this genre, it's OK. Only worked through about half of this and
then skimmed the rest, so I may revisit this later if the need/desire arises.
*** DONE Essentials of the Java Programming Language
- State "DONE" [2010-03-24 Wed 02:17]
- State "STARTED" [2010-03-23 Tue 23:48]
Too basic. This book is apparently for people almost entirely new to
programming.
*** DONE Everything you need to know about cryptography in 1 hour (lecture notes)
- State "DONE" [2010-05-14 Fri 21:14]
- State "STARTED" [2010-05-14 Fri 20:59]
Slides e-book. OK, but not enough detail on why recommended stuff is better
than the alternatives to be useful.
Notes:
- Recommends SHA-256 for hashing, HMAC-SHA256 for symmetric authentication,
AES-256 for block ciphers, and CTR for block cipher mode of operation.
- Recommends RSASSA-PSS and a 2048-bit RSA key with public exponent of 65537
and SHA256 for asymmetric authentication.
- Recommends RSAES-OAEP and a 2048-bit RSA key with public exponent of 65537,
SHA256, and MFG1-sha256 for asymmetric encryption.
*** DONE Coders At Work
- State "DONE" [2010-05-19 Wed 01:02]
- State "STARTED" [2010-05-02 Sun 04:39]
Since I have to listen to boring stories about people's jobs at work every time
I interview someone, just sticking to the ones who've done interesting things
or are likely to have interesting things to say. Reading: Jamie Zawinksi,
Douglas Crockford, Brendan Eich, Joe Armstrong, Simon Peyton Jones, Peter
Norvig, Guy Steele, Ken Thompson, Donald Knuth.
This book can be a great read sometimes, but there are mountains of
uninteresting text between the good parts. Skipping the answers to questions
that are too personal (like questions about their first programs and such)
tends to shorten the read by about half.
I'm not sure every one of the guys interviewed here are really the best picks.
There's many more living legends out there and brilliant theoreticians, yet
we're sometimes stuck reading about some guy who wrote a CMS in PHP. Others
have done a couple interesting things, but we have to listen to their opinions
on stuff they obviously haven't thought about much. Of course, this is more
about "coders" and less about computer scientists, so the book delivers on its
promised content.
Some interesting observations:
- Most of these guys hate C++ and IDEs. They also generally shun OOP.
- Most use very primitive debugging techniques, like simple printf statements.
- One problem I've always noticed is how to walk into a huge codebase and grok
it fast. Apparently, these guys struggle with that too, and there's no
method that really makes this easier. The most common method is to just pick
out operations you understand as a user and walk through the code to figure
out how its done. Eventually, you start to develop a mental model.
- Most don't really do any formal SDLC-type design work. Most of it is either
rapid prototyping or just mental visualization combined with rewriting.
*** DONE JavaScript: The Definitive Guide
- State "DONE" [2010-05-19 Wed 02:00]
- State "STARTED" [2010-03-16 Tue 10:03]
Read a little, but this book is needlessly huge. I've been writing JavaScript
for a while now and think I have it down for the most part. Will just leave
this one in the office as a reference. The book JavaScript: The Good Parts
seems to cover all the important stuff.
Might try reading this one instead, if I find myself needing a bit more:
http://www.techotopia.com/index.php/JavaScript_Essentials
*** CANCELED Design Patterns
- State "CANCELED" [2010-06-08 Tue 10:57]
The eponymous Gang of Four book. Already skimmed various parts of this, but
since they come up in SE all the time, might as well finish it.
Update: Sold this to a coworker for $20, so canceling this task. I still have
it on e-book though, if I really want to read it.
*** DONE Big Ball of Mud (paper)
- State "DONE" [2010-06-08 Tue 20:26]
- State "STARTED" [2010-06-08 Tue 19:51]
Online at: http://www.laputan.org/mud/mud.html
Stopped reading about 1/3rd in since it wasn't saying anything I didn't already
know.
*** DONE JavaScript: The Good Parts
- State "DONE" [2010-06-26 Sat 23:08]
- State "STARTED" [2010-04-21 Wed 09:51]
Good book. Makes one wish books of this quality existed for every language.
Chapter 2's flow diagrams are rather useless, however (BNF definitions would be
more useful).
Took notes in ~/src/repo/javascript/jstgp and added some useful stuff mentioned
throughout the book in ~/src/repo/javascript/utils.js.
Skipping JavaScript regexps, due to it having it's own custom syntax for it and
that I don't want to memorize a bunch of alternatives for doing the exact same
thing. I'll just Google that when needed. Chapter 8 contains a handy list of
built-in methods.
Of course, writing real JavaScript involves a lot of ugly things this book
doesn't cover, like the DOM and its properties, memorizing all the built-in
utility functions, and using libraries. I find the level of crap acceptable
using it as a general programming language, but with all that added by these
things, it's still pretty miserable.
This book really brought me from knowing next to nothing about JavaScript to
being at least competent in it. It still doesn't feel completely natural to me
though, but that's mainly because I haven't written enough code in it. I'm
sure that will come with time if it's really needed. My only other comment is
that while I really appreciate the patterns presented in this book, the
language itself makes these rather cumbersome to implement, so I'll probably
try offloading as much as possible to lower tiers and maybe a little to
frameworks like jQuery.
*** DONE Core Java, Volume 1: Fundamentals, 8th Edition
- State "DONE" [2010-07-14 Wed 17:13]
- State "STARTED" [2010-06-26 Sat 23:16]
A painful read, but gets the job done about as well as can be expected. Only
read chapters 1-6 and 10-13. The goal here was just a refresher of previously
learned Java language features/syntax, plus taking stock of the features added
since 1.2. Since I read much of the 2nd edition of this same book back in
college, was able to speed read it.
Setup notes:
- To install the library source into a JDK install, grab the src.zip file and
make a src directory in JDK home. cd to it and run `jar xvf ../src.zip'.
- To install the docs, grab jdk-version-doc.zip, put it in JDK home and while
there run `jar xvf jdk-version-doc.zip'.
- The corejava8.zip file is in ~/src/java and extracted there too.
- Entry point to the JDK docs on FreeBSD is at:
file:///usr/local/diablo-jdk1.6.0/docs/api/index.html
Notes:
- Use the appletviewer app to test applets outside of a web client, sending it
the HTML file.
- Use /** */ for block comments, as these nest properly.
- Makes a case for conventions (which I think I'll adopt) of putting instance
variables at the end of a class definition and always using `this.' when
referencing them within a class.
- Always use .clone() when returning a mutable data field in an accessor
method. E.g.:
public Date getHireDate() {
return (Date) hireDate.clone();
}
- `main' methods can be added to any class and are a good way to do ad-hoc
unit-testing.
- 1.6 adds the ability to include in CLASSPATH all .jar files in a directory by
using this syntax (Note that the quotes aren't necessary on Windows):
CLASSPATH=$CLASSPATH:~/lib/archive/'*'
- Valid Javadoc tags: @param variable description, @return description, @throws
class description, @author name, @version text, @since text, @deprecated
text, @see reference.
- See chapter 4's Javadoc section for more Javadoc specifics if ever generating
full Javadocs for a particular app/library.
- When creating equals() methods, the parameter should be of type Object, check
to see if they are the same reference, check if the other is null, check if
other is an instanceof this class, cast other to this class, then do normal
fields checking. See chapter 5's equality testing section for details.
- Java collections cannot accept primitive types as type parameters. Automatic
boxing and unboxing occurs with collection method parameters/return values.
- Java can define methods to take arbitrary numbers of parameters, like this:
public int sum(int... values) {
int sum;
for (int i : values) { sum += i; }
return sum;
}
- The `jar' command supports creation of jar files with manifests that specify
"Main-Class:" which is the default class to look for the `main' method in.
- If doing a serious app, use the Preferences API. This has advantages like
being platform-independent and providing the ability to import/export them.
For smaller stuff, use java.util.Properties.
- When throwing a new exception in a catch block, create the new exception
object, then call `newex.initCause(e);', where e is the original exception.
This allows for tracing through nested exception stacks.
- When doing industrial-strength logging, create your own logging.properties
file and specify it at with `java -Djava.util.logging.config.file=myfile'.
Otherwise, jre/lib/logging.properties is used.
- To debug classpath issues, launch the JVM with the -verbose flag.
- `javac' has a built in lint option. See `javac -X' for details.
- Check out the GUI app `jconsole' for runtime monitoring of a Java app.
*** DONE How to be a Programmer
- State "DONE" [2010-07-15 Thu 19:16]
- State "STARTED" [2010-07-14 Wed 17:14]
http://samizdat.mines.edu/howto/HowToBeAProgrammer.html
While most of this is okay advice, this was a boring read.
*** DONE Engineering Large Projects in a Functional Language (paper)
- State "DONE" [2010-07-27 Tue 09:01]
- State "STARTED" [2010-07-27 Tue 08:50]
A slide series from Galois, Inc. Not very informative on its own, but has a
few good tips.
*** DONE CSEP: Programming Languages (lecture series)
- State "DONE" [2010-09-07 Tue 10:11]
- State "STARTED" [2010-07-29 Thu 01:01]
Has a lot of Caml stuff in it, but also spends several lectures on lambda
calculus, type systems, writing interpreters, and a few other topics of
interest. mp3s downloadable from:
http://www.cs.washington.edu/education/courses/csep505/06sp/lectures/
Like other attempts with using an audio format for learning complicated topics,
this works considerably less effectively than text. But, if you're stuck in a
car, it's better than nothing, and in this case, I'm pretty familiar with the
concepts here already. The syllabus here is well chosen also, unlike most
programming language courses.
*** DONE UNIX Power Tools
- State "DONE" [2010-10-10 Sun 21:11]
- State "STARTED" [2010-10-05 Tue 12:56]
As suspected, I already knew the vast majority of the content here. However,
since I use Unix all day, the few things found here will pay off for the time
spent reading it. Skipping a few parts that don't interest me, like stuff
about non-Bourne shells, old VCSes like RCS, tape drives, vi, sed/awk (what
appears here is mainly just snippets from O'Reilly's Sed & Awk, so I'll read
that on it's own sometime), etc.
Notes:
- Some useful utilities to keep in mind: cut, comm, sleep, fc, tpipe, od, hd,
and fmt.
- inodes contain data about files. There is one inode per file. `ls -i' will
list the inodes for files. The `stat' command will list the contents of a
file's inode, which include things like its ownership, timestamps, size, etc.
- You can background a running process by suspending it first with C-z and then
running `bg'.
- Modified prompt in root's .cshrc to be red in reverse video with the full
working directory path.
- Redirection operators:
stderr to a file: prog 2> file
stdout and stderr to a file: prog > file 2>&1
stderr to end of file: prog 2>> file
- here documents let you input file data from stdin without creating a file.
Here's an example, creating a file with 2 lines:
% cat > testfile <<test
heredoc> sdf1
heredoc> sdf2
heredoc> test
- Building strings with {}: This pattern-expansion syntax allows you to specify
a list of changes to expand a single expression to. For example, `mv
file{1,2}.c' will expand to `mv file1.c file2.c'. If not sure, test these by
echoing them first. One useful tip is to use `cp file{,.bak}' to quickly
backup a file.
- The foreach control block does the same thing as in normal languages. This
works in csh, tcsh, and zsh:
% foreach file (file{1,2,3})
foreach> touch $file
foreach> end
In bash, use the for loop:
$ for file in file{1,2,3}
> do touch $file
> done
- Using find's -exec flag can save an xargs call, e.g. `find . -name 'file*'
-exec rm -i {} \;'.
- The !:n* (where n is an integer) construct in bash/zsh might have a use.
This expands to the nth through last argument in the previous command,
e.g. `cp file1 file2' followed by `echo !:1*' expands to `echo file1 file2'.
- The ^^ construct allows a replacement within the last command, then it is
repeated, e.g. `echo hi' followed by `^i^ello' expands to `echo hello'.
- !-expansions can be checked with a :p suffix. Then this can be followed
with a `!!' to actually run it.
- `kill' can reference job numbers (shown in `jobs'), e.g. `kill %1'.
- The subshell construct () can be used for collecting output from stdout and
stderr into a single pipe and for a series of commands that need to change
the environment to run without changing the parent shell's environment.
- Use find's -o and -a operators to combine search parameters, e.g. `find . \(
-name file1 -o -name file2 \) -print'.
- tar's z option can also be used when creating gzipped tarballs, e.g. `tar zcf
stuff.tar.gz stuff', where stuff is a directory.
- `sdiff' is a side-by-side version of `diff', but it requires a >80 column
terminal.
- One use for `exec' is to replace the current shell process with a different
program. This can be handy if one of the programs started in this manner by
my screen rc file dies and I want to restart it without creating another
wrapper shell process and have it lie around, wasting resources.
- When running commands where the default umask (027 in my case) is
inconvenient, like modifying a Darcs repo, a subshell can be spawned, wherein
the umask can be changed without affecting the parent shell.
- Use the `time' command to log the runtime of a process.
- There's an ASCII lookup table in /usr/share/misc/ascii.
Learned a lot of useful tricks here, but many aren't that life-changing in my
case, since I use Emacs for complex editing instead of filters, screen instead
of job backgrounding, org-mode for office automation, and scripts in real
languages instead of shell. Overall, this is an okay book and worth reading,
but includes a lot of filler sections just describing how to use some flag for
a program and spends way too much time on file editing. It would have been a
much more useful read to me some 10 years ago.
*** DONE Maven: The Definitive Guide
- State "DONE" [2010-11-01 Mon 13:25]
- State "STARTED" [2010-10-22 Fri 11:04]
Just reading the intro, part I, and the first few chapters of part II. The
rest is mainly intended as a reference.
Notes:
- Maven uses a bunch of conventions, like ${basedir}/src/main/java for code,
${basedir}/src/main/resources for resources, and ${basedir}/src/test for unit
tests.
- Check http://mvnrepository.com/ for versions and XML snippets.
- To create a new project, use something like: mvn archetype:create
-DgroupId=com.macroexpand.mygroup -DartifactId=myproj
-DpackageName=com.macroexpand.mypkg
- Other archetypes exist for other types of apps, like webapps and Swing apps.
- Use `mvn help:effective-pom' to see the aggregate effective POM.
- List of all maven.org packages at: http://repo2.maven.org/maven2/
- `mvn site' generates documentation and reports.
- The pom.xml file can contain license and developer information. See 4.4 for
details.
- After installing stuff to the local repo, use something like this to run it
`mvn exec:java -Dexec.mainClass=com.macroexpand.mygroup.myproj.Main
-Dexec.args="my_cli_args"'.
- Get the full description of plugins using this form (the exec plugin is shown
here) `mvn help:describe -Dplugin=exec -Dfull'. In this example, it shows
all the valid parameters that could be used in the previous exec call.
- View the project dependencies in list form with `mvn dependency:resolve' or
tree form with `mvn dependency:tree'.
- Stick unit test case data in src/test/resources.
- To package and distribute a project and its dependencies as an assembly use
the maven-assembly-plugin with a descriptorRef of jar-with-dependencies and
run `mvn install assembly:assembly'. See 4.13 for details.
- There's a maven-tomcat-plugin that could be useful for webapps.
- SNAPSHOT is a special version in Maven that indicates the latest code,
typically TRUNK or HEAD.
Keeping this book around for reference purposes. There's a lot of extra topics
here, covered in detail, that I skimmed or skipped in order to lazy load. See
Maven under "Work-related topics" for more notes and some commentary.
*** DONE Augustus/PMML documents
- State "DONE" [2010-11-01 Mon 14:25]
- State "STARTED" [2010-09-23 Thu 12:28]
Reading whatever I can find on Augustus and PMML.
Papers:
- Augustus: The Design and Architecture of a PMML-Based Scoring Engine: A short
paper from 2006.
- Augustus Documentation Version <IP_ADDRESS>: Keep this as a reference in case I
need to know what goes where.
Open Data Group slides:
- Analytic Models:
- There are four basic data mining algorithms: nearest neighbor algorithms,
k-means clustering, classification and regression trees, and CUSUM.
- k-means clusters converge on the centroids (geometric centers) of final
clusters.
- PMML-Based Analytic Architecture: Just an overview of PMML.
- Introduction to Augustus: A decent overview. Could be handy for defining
producer input later.
- Case Studies: Tree Model, Naïve Bayes Model, Highway Traffic: Just some
examples.
- Case Studies: Change Detection in Baseline Models: Has an overview of SCADA,
among other things.
*** DONE Repository Management with Nexus
- State "DONE" [2010-11-09 Tue 16:14]
- State "STARTED" [2010-11-09 Tue 09:53]
This is the official Nexus book from Sonatype. Nexus features I'm interested
in are: using it as a central repository and integration with m2eclipse. Only
reading chapters 1, 2, and 4.
The link to the PDF of the latest version appears broken, so reading the
crappier HTML version: http://www.sonatype.com/books/nexus-book/reference/
Notes:
- There's two versions of Nexus, Open Source and Professional. The OS version
offers everything I'll never need.
- Chapter 4 says to have Maven use a Nexus repo, one must edit
~/.m2/settings.xml, but at work, the pom.xml files just have repository
blocks pointing to a Nexus URL like:
http://<server>/nexus/content/repositories/releases
Nexus is okay in that it occasionally fills a needed role in some development
projects. But, it's also overly complicated and extremely boring to read
about. If I ever miss reading about boring software in the future, I can also
check out Hudson, which is often combined with Nexus and Maven.
*** DONE RESTful Java Web Services
- State "DONE" [2010-11-12 Fri 13:41]
- State "STARTED" [2010-11-10 Wed 12:43]
Reading chapters 1-4, and 7. While it covers RESTEasy, it dedicates most of
the book to the other Jax-RS frameworks of Jersey, Restlet, and Struts, which
I'm ignoring. The reviews of this book are pretty bad, so just giving it a
fast read.
Notes:
- REST is just a set of constraints that define a system: client/server,
stateless, supports caching, uniformly accessible, layered, etc.
- HTTP methods: POST (create), GET (retrieve), PUT (update), DELETE.
- Created a Maven project for a Twitter client in ~/src/repo/java/resttwit.
This app endlessly scrolls garbage from Twitter in near real time to the
console. After writing it, I stared at the moronic "thoughts" of the
cesspool that is humanity for a few hours.
- The RESTEasy framework is pretty simple to use. Use chapter 7 as a
walkthrough while writing one if I ever have to.
- For RESTEasy, basically, you just define a class for each URI. Within that
class, you add a few annotations: @Path maps the class to the path (sometimes
including parameters as extensions to the URI in braces), @GET/@POST/etc. map
methods to HTTP request types, and @Consumes/@Produces map MIME types to
methods.

This book is indeed pretty low quality and sloppy. If I want to read up on