amXor

Blogging about our lives online.

Showing posts with label Programming. Show all posts
Showing posts with label Programming. Show all posts

11.24.2010

Chaining in JavaScript

0 comments

It's been bugging me for a while that I didn't understand the method behind jQuery's function chaining. So I built the smallest functional version of it. I call it aQuery (for andy, or awesome).

This is the calling code:

window.onload = function(){
    aQ("#content").height().background().elem.innerHTML = "hello";
};

I have made two functions .height() and .background(), which do pretty much what their names suggest. The elem is the original object and so I use it to call "native" code. Maybe should have called it "target" to be consistent.

Here's the meat and potatoes of the "library":

(function() {   
    var aQ = function(selector) {
        return new aQuery.fn(selector);
    };

    var aQuery = {};
    aQuery.fn = function(selector) {
        this.elem = document.getElementById(selector.split("#")[1]);
        this.cssText = "";
        this.height = function(){
            this.cssText = this.cssText + "height: " + window.innerHeight + "px;";
            this.elem.style.cssText = this.cssText;
            return this;
        };
        this.background = function(){
            this.cssText = this.cssText + "background-color: #444;";
            this.elem.style.cssText = this.cssText;
            return this;
        };
        return this;
    };
    
    return (window.aQ = aQ);
})();

Each call to aQ makes a new object, primed with the selector string. This object has several functions and attributes, and all the functions return "this", so they can be chained together. As it stands, it's not dynamic at all but you can use your imagination!

11.17.2010

jQuery Mobile Data Attributes

0 comments
In testing out the newly released jQuery Mobile library, I'm left with strong but mixed feelings about it.

First, it's a surprisingly unique solution. The jQuery team is very good at simplifying hard problems, but in some ways you come to expect that the solutions will be quite similar. I expected a bunch of jQuery-type functions that would transform your markup into the mobile look and feel. I expected jQueryUI for mobile, and was ready to start moaning about how inflexible and unsemantic it was.

Instead, their solution is to add HTML syntax and automatically wire up all of the functionality for you. Not what I was expecting. You don't have to write a single line of JavaScript for the default functionality. This has got to be the easiest solution for porting HTML content to a device. Your "pages" can all live inside one HTML file or be loaded dynamically with AJAX. All the URL's get dynamically referenced to the main page and resolve correctly. And no handwritten JavaScript or CSS in sight.

A lot of this functionality comes through the use of custom data attributes in your HTML. In my perusing of the HTML5 features I have to admit: this one seemed fairly boring. So what if I can add custom attributes and my page still validates? Why should I care?

But the guys at jQuery Mobile have capitalized heavily on this notion. By using custom attributes quite liberally to alter the semantic meaning of lists, headers and anchors, they can automate all the behind the scenes JavaScript and CSS necessary to make it work.

It's a strange reversal of the web technology where the withering old HTML document is front and center again and the CSS and JavaScript hide in the back room, pulling on the invisible strings.

For a "Navigate to Text Content" site, this is a dead-simple solution. I think a 'Project Gutenberg' site is at the top of my list. Or maybe an RSS reader.
Some negatives that I see:

  1. Building or altering UI components is going to be a pain.
  2. Really only scales to iPhone-size devices. (They support iPad, but who uses full-width buttons on iPad?)
  3. The UI is a tad bland. (Erring on the side of caution, I guess)
  4. I'm undecided on all of those data-role tags.
  5. It's new and strange. I don't like new things. I'm scared of it. What if it doesn't like me?
Anyway...

My real takeaway is not the tool itself, but understanding how it works and how I can incorporate those ideas into my own projects. There's a lot of smart people working on projects like this, and you really only get to know them by reading their code.

7.17.2010

Applescript Mindbender

0 comments

This is a crazy hack!!!

http://www.macosxhints.com/article.php?story=20060425140531375

How to have the same file contain both a shell script and a compiled AppleScript

% /usr/bin/osacompile -o shebang.scpt
on run argv
    argv
end run
^D

% chmod +x shebang.scpt
% echo \#\!'/usr/bin/osascript' > shebang.scpt
% ./shebang.scpt a b c
a, b, c

This puts a compiled applescript in the resource fork and a shell script in the data fork... mind exploded!

Why The Command Line?

0 comments

Vaporware

The software arms race is heating up. At the forefront are apps, which are small, single purpose programs that are very stylish, but generally have very little functionality. The reason I say that they have very little functionality is because they are built as small standalone, self-contained programs designed for a very specific purpose.

The iPhone needs multitasking because every little bit of functionality is wrapped in it's own app. The average computer user has many windows open and many applications installed because we need many apps to complete one task.

In all these things, I believe that software developers have lost their way. The user is forced to multitask extensively and perform every task one step at a time.

The Killer App

For those that do not understand the UNIX command line, it looks like a deviously complex and foreign language. There are no visual clues about how to accomplish a certain task, no tooltips or interactive dialogs to walk you through.

What they don't see is thousands of purpose-built apps all designed to work seamlessly with one another and to be completely automated. Using a command line is not just a bunch of typing to achieve the same result as a mouse command line, it is a bit of typing to achieve what a thousand mouse clicks could do.

The standard question I get when people see me working in a fullscreen terminal is "Wow, are you a hacker?" No, I'm not a hacker in the sense you mean. I'm not decyphering encrypted passwords and finding loopholes in government databases or anything like that. I am a hacker in the sense that I'm not constrained by the functionality offered by graphical interfaces. I am a hacker in the sense that if I have repetitive or menial tasks, I'd rather let the computer do the work for me.

Learning the basics of the command line takes some work. Learning how to use the powerful text editors like Vim or Emacs takes some work. Learning how and when to use the versatile processors like Sed and Awk takes work. Learning how and when to tie these commands together into shell scripts or functions takes some work. But in all these things, it is work that pays off in higher productivity and far fewer hours spent repeating tasks that are better automated.

But the true value of the command line comes when you can effectively monotask. When you can focus on the task at hand and automate all the housekeeping. And when I say "housekeeping" I probably mean a lot more than you think.

For example, here's my current workflow for blogging:

  1. blog why_the_command_line? >> opens a new file in my blog folder in the Vim text editor.
  2. Type post using vim and the simple Markdown formatting.
  3. %s/CLI/command line/g >> Replace all occurrences of 'CLI' with 'command line', cause I knew I'd be typing it a few times.
  4. "publish why..." >> run the file through the Markdown to HTML formatter and email it to my blogger page.

I'm using two simple scripts here, one called blog and another called publish, and for most workflows that's all it takes. I now have a local copy in the easy to read markdown format and one on my blog in HTML format.

The value in this is partly the shortcuts, yes, but mostly the lack of context switching. There is no need for multitasking, because everything I need is a few keystrokes away, which allows me to focus. And that's a good thing.

Building Toolchains

0 comments

A toolchain can be a very subtle thing.

We make them all the time, a lot of times without even realizing that we are doing it and very often without realizing that there might be a better way. In order to really use computers effectively, we must pay attention to the toolchain, or workflow that we are using.

A simple example could be task lists. Your toolchain could be as simple as: "Every todo list goes on a Sticky note". This works but it isn't terribly versatile. You have to manually clean them up when you are done with them, you can't access them from your phone or another computer, and there's no way of sorting them by date unless you sort them visually by placement.

This could be improved by using a spreadsheet or database like Excel or Bento. Then you have the option of adding priority, type and date fields and sorting them accordingly. You could then use Google Docs and make this spreadsheet accessible online. It's definitely better, but in my opinion it's a bit over the top just for a todo list.

My established toolchain is the task list in Google Calendar. It's simple; the only necessary info is a title which gets a checkbox beside it to check when the task is done. You can optionally add a due date, notes about the task and you can move them to another list or delete them when finished.

But the new toolchain that I'm playing around with is even simpler, it's simply a private Twitter feed. Instead of being limited to todo's, I post anything that I want to remember. I can delete the posts if it's a completed task or I can simply let it fade into the timeline. I can retweet if it's something important that might be getting stale and I can include links if it's related to something on the web.

This is a simple example, but I think it highlights a few important ideas. Anything you do with a computer can probably be done tens or hundreds of different ways. With that in mind, is your current toolchain the best for you? Some things to keep in mind when building a toolchain:

  • Is the content properly accessible (over time and space)?
  • Does the content need the draft/publish distinction?
  • Is there a single endpoint (don't repeat yourself!)?
  • Is there a concern about losing your data to a cloud service?
  • Do you need a revision history?
  • Is there other related tasks that can use the same toolchain?
  • Is the endpoint shared, collaborative or private?

Many applications, including most cloud services, manage the data for you. This can be convenient for simple tasks, but can be troublesome if you want to extend or customize your toolchain beyond what the service offers. It's wise to think for a moment whether such programs should be the single home to your content, or whether they should be an endpoint of a larger toolchain.

When I started using blogger, I used the online editor, publishing as I went. After a little while I thought, "Hmm, it would be nice to have a local copy of my blog." It is possible to export your entire blog, but this is not practical on a day to day basis. The better solution is to use the cloud service as an endpoint, not the only point. By working on a local copy of my blog and publishing it to Blogger, I'm able to do some things that I can't with the Blogger tools. I now use Markdown formatting to simplify markup, and I can archive this version with version control using Git.

It's not a complex toolchain, but it's nice to know that I have a platform neutral copy of my blog that I can reuse however I see fit.

7.15.2010

Thought Experiment: JavaScript POSIX Shell

0 comments

Okay, this one is a bit of an odd idea. My thinking is that you build a shell that will work in a browser window. The 'filesystem' is created in the browser's localStorage. Only the basic IO system and OS facilities are hardwired, the rest is implemented through executable 'files' in the filesystem.

The general idea is to make a lightweight shell that is programmable and could do some really cool stuff.

echo "Hey, I'm in a browser terminal!" | twitter

I've already hacked together a demo just to see how all the parts would work. Here are my findings.

User Interface

A text area is editable or not, whereas a terminal window is only editable at the cursor point. I puzzled over this one for a bit, wondering how to mimic this behavior in a browser, and then I dropped it and just did what was convenient: one editable textarea for commands, one for output. This is what that looks like:


At first I thought this was kind of a lame hack, but as I use it a bit more it feels quite intuitive. It's like posting on Twitter; you type on top and the command gets pushed onto the stack. But, you might ask, what about programs that take over the full terminal window? Well, in this case you just pass the lower window to the program and keep the upper for commands to the program. In the Vim editor, and most others, you always have a status/command line used for issuing commands anyway, so why not keep it on top?

Calling Convention

This is the one place that I wanted to stay very UNIXy. The commands are called with arguments and have access to stdin, out and err and everything is a 'file', meaning everything is a key/value pair in localStorage.

Files

In my first draft, the files are just key/value pairs with the file name being the key and the text being the value. There are a few problems with this.

First, file permissions/types are not implemented, which is mostly a problem because there needs to be a mechanism to determine what are executable files. Remember: everything is a file; all the commands 'echo, emacs (!)' will be JavaScript strings stored in the browser. Any user can make their own executables.

Second, a file heirarchy would be nice. Timestamps would be nice.

I think the best solution is to format the 'name' string with all the metadata, which would get stripped when accessed by the Terminal. So a file like /bin/echo would become < name='echo' path='/bin/echo' mode='-r-x' > if we want to use XMLish formatting.

Summary

I really like this idea. When I first got it up and running I was a bit giddy. It is browser based, but I also want to make it very Internet based. I would like to make it possible to wire in any site that has a public API; Social Networking, Mail, RSS, the works.

Skratch Development Update

0 comments
"Make one to throw away"

I forget who said this, but it has been good advice for my little web-based sketching app called Skratch. After the first version was up and running I took a step back and had a hard look at the overall structure.

Overall it wasn't too bad. It has been a work in progress to figure out how to modularize and encapsulate. In the alpha stages it was a bunch of global functions and variables going every which way. By version 0.1, I had realized that I needed to introduce some structure and so I grouped the drawing code into objects with attributes. Most of the event handling code was still in global functions. At this point I took a break and did some reading.

First I read Douglas Crockford's JavaScript: The Good Parts, and none of it really sunk in. It's an excellent book for the theory of how to use the language functions and how to avoid bad practices, but it doesn't really give the reader a sense of how they should write JavaScript. All of the examples are very short and specific to a certain language feature.

Then I read the jQuery source code and realized that I hadn't got it at all. It took me a while to really grasp what was going on here, and most of that time was spent trying to figure out what it was that I didn't understand. The thing that I didn't understand was simple but elusive: closure.

I have heard of closures as a programming concept a number of times but I never really grasped why they would be practically useful or how I would recognize one when I saw one. Reading the jQuery source, it took me a while to realize that this was a ~6000 line closure staring me right in the face. So I started coding to see how I could use an anonymous function to my benefit.

It's really quite simple: any vars, functions or objects declared inside a function are only accessible within that function. So if an anonymous function invokes itself, it's totally possible to not add to the global namespace at all. Nothing within that function is accessible unless it is attached somehow to the global namespace. But those functions that ARE attached to the global namespace can make use of all the private variables and state within the anonymous function.

This is the little testing function that I hacked together to prove this to myself:
(function () {
    var obj = {}, 
        privateData = 42,
        privateMethod = function (arg) {
            return "Private method: called with "+ arg + 
                    "\n" + "The holy grail: " + privateData + "  sees:" + obj.globalData;
        };
    
    //Priveledged: public methods which rely on private members.
    obj.protectedMethod = function (arg) {
        return "Priveledged method: called with " + arg + "\n" +
                    privateMethod("really " + arg);
    };
    
    // Public: members that are publicly accessible, rewritable, etc...
    obj.globalData = "I am the viewable data of the globalobject.";
    
    obj.generateString = function () {
        return "Global method: " + obj.globalData + "\n" +
                obj.protectedMethod("do this");
    };
    genericObj = obj;
})();

Only in the last line do I "publish" the object I've created, anything else is encapsulated in the closure and remains private.

Once I grasped this concept, it was immediately apparrent how it would be useful to my drawing app. I created an object, called "skratch" and attached only the functions and data that need to be accessible from outside to it; basically just the UI controls and a .setupCanvas method. The drawing engine and event handlers are now completely wrapped inside the closure.

This way of programming seemed kind of ugly to me until I looked at the DOM tree for my app. There was only one global object called "skratch" with a couple methods for triggering UI events and that was it.

So, I now know what closure is and what it can do for me and I think my scripts will be better for it.

Oh, and FYI, after reading the jQuery source, I couldn't help but include it in my app, even though all it does right now is make the toolbar go swoosh! Totally worth it!

6.09.2010

Skratch - HTML5 Version

0 comments

I've rewrote my drawing engine to run entirely in javascript and render to an HTML canvas. Since all the control logic was already sketched out and working in Quartz Composer, I simply had to translate that into javascript equivalents.

Instead of drawing with a colored image, this version draws individual pixels for every point along the path with a random scattering algorithm. It does this by computing a random angle and a random distance from the drawing position.

The controls are fairly rough at the moment. I think I'll probably scrap all the HTML inputs and integrate the controls into the drawing surface itself; but this works for testing purposes.

The background is drawn using tips from this great tutorial.

The code is in one HTML file, which you can download and run locally. Get it here, In the folder "Web Development" > Pencil_Drawing.html.

6.03.2010

Skratch Rendering Engine

0 comments

Abstract

Skratch is a Quartz Composer based rendering engine for painting and drawing applications. The primary design considerations are speed, smooth dynamics, a simple interface and extensibility.

Features

  • Quadratic Input Smoothing
  • 3 Mode, Fullscreen Color Chooser
  • Pressure Sensitive Brush
  • Selectable Brush Types and Repeat Rate
  • Transparent Drawing Layer
  • Modular Design

Design

Skratch is designed to be highly modular in order to make modification, automation and extensibility simple.

Smoothing Functions

One of the primary design obstacles was the relatively slow sampling rate of the mouse or tablet (approx. 16.7 ms). This results in jagged lines at even moderate brush speeds. To overcome this limitation I have implemented a smoothing algorithm using a quadratic Bézier curve in Javascript. The function takes three sampling points input(3) from the mouse. It then calculates three control points, one halfway between in0 and in1, one at in1, and one halfway between in1 and in_2. The Bézier curve is then calculated based on these points and the number of intermediate points is determined by the "Granularity" input. The smoothing function looks like this:

(((1-t)*(1-t)) * p0) + ((2 * (1-t)) * t * p1) + ((t*t) * p2)

Where t is the point to produce in the range 0.0 - 1.0 and p0, p1 and p2 are the control points. http://en.wikipedia.org/wiki/Bézier_curve

On top of this rapid smoothing, there is also the option of low-speed interpolation which makes it much easier to draw smooth curves and lines by hand. This makes it easy to switch between quick sketching / hatching and smooth deliberate strokes.

Modularity

The original goal was to develop a simple sketching program, but due to it's modular nature, it has been quite simple to connect it to function generators and produce auto-drawings with interesting results. This modularity should make it quite simple to extend it beyond Quartz Composer in XCode and produce various types of drawing programs.

Color Chooser

The color chooser that I've implemented is somewhat different than most. When you press the 'c' key you are presented with a full-screen CMYK palette to pick a color from. If you use the right arrow key, it will change the palette to a number of alternate palettes. There's also an RGB palette as well as an option to pick the color right from the current drawing (in a slightly pixellated version). Further images or palettes could be added with minimal effort.

Transparency

The drawing layer is transparent, allowing full color drawing and erasing and the potential of stacking layers with blending. The background is opaque.

Further Directions

  • Canvas Sizing (Zoom & Pan)
  • Custom Brushes & Brush Selector
  • HUD of all drawing state info

Noodling.

Guided Auto-Drawing.

3.08.2010

Arbor Redundancy Manager

1 comments

** UPDATE! This code will potentially create false hard-links between files that are very similar. I suddenly have many iTunes album covers that are from the wrong artist. I'm not sure if this is a result of how cavalier Apple is about messing with low level UNIX conventions, or if my code is faulty. Beware! **

We all have redundant data on our computers, especially if we have multiple computers. My new tool is aimed at making backups for this kind of thing simple.

I'll take the simplest practical example: You have two computers and one backup hard drive. You want full backup images of both of these systems. Even if you're just backing up the 'Documents' folder, there is a good chance that there is a lot of redundant files. What my little program does is checks the contents of these files, and if two are the same, it creates a hard link between them.

So, if your backup folder looks like this:

Backup
Backup > System 1 > Documents > somefile.txt
Backup > System 2 > Documents > somefile_renamed.txt

It will look exactly the same after, but there will only be one copy of the file, if the contents are identical.

I have dozens of backup CD's that contain a lot of the same information, now I don't need to sort through them and reorganize or delete duplicate files. I can leave them just as they are and any duplicate files will be linked under the hood. Here is the current state of the code, discussion to follow:

arbor.py - v. 0.02

#!/usr/bin/env python
import os, sys, hashlib

arg_error = False
if len(sys.argv) == 2:
   src = sys.argv[1]
   srcfolder = os.path.abspath(src)
   if not os.path.isdir(srcfolder):
       arg_error = True
else: arg_error = True

if arg_error:
   print "Usage: arbor [directory]"
   sys.exit()

backupfolder = os.path.join(srcfolder, ".arbor")
if not os.path.isdir(backupfolder):
   os.mkdir(backupfolder)

skipped_directories = [".Trash", ".arbor"]
skipped_files = [".DS_Store"]
size_index = {}
MAX_READ = 10485760

def addsha1file(filename, size):
   if size > MAX_READ:
       f = open(filename, 'r')
       data = f.read(MAX_READ/2)
       f.seek(size/2)
       data = data + f.read(MAX_READ/2)
       sha1 = hashlib.sha1(data).hexdigest()
   else:
       sha1 = hashlib.sha1(open(filename, 'r').read()).hexdigest()
   backupfile = os.path.join(backupfolder, sha1)
   try:
       if os.path.exists(backupfile):
           os.unlink(srcfile)
           os.link(backupfile, srcfile)
       else:
           os.link(srcfile, backupfile)
   except:
       print "Unexpected error: ", sys.exc_info()[0], sys.exc_info()[1]

fcount = 0
for root, dirs, files in os.walk(srcfolder):
   for item in skipped_directories:
       if item in dirs:
           dirs.remove(item)
   for name in files:
   fcount += 1
   if fcount % 500 == 0:
       print fcount, " files scanned"
       if name in skipped_files:
           # print name
           continue

       srcfile = os.path.join(root, name)
       size = os.stat(srcfile)[6]
       if size_index.has_key(size):
           if size_index[size] == '':
               addsha1file(srcfile, size)
           else:
               addsha1file(size_index[size], size)
               addsha1file(srcfile, size)
               size_index[size] = ''
       else:
           size_index[size] = srcfile

Here are the added features of this version:

  • Folder to backup is passed as command-line argument
  • Backup files are placed at the top level of that folder in the .arbor directory. These are just hard links so they don't really add any to the size of the directory.
  • Ability to skip named folders or files
  • Only calculates a checksum if two files are the same size.
  • Only calculates a partial checksum if a file is over 10mb. Checks 5mb from start and 5mb from middle.
  • Prints a running tally of files checked (per 500 files)
  • Doesn't choke on errors: some files don't like to be stat'ed or unlinked, permissions issues.

I'm not sure about the partial checksum option, but it was really bogging down on larger inputs. It's not really practical to do a SHA1 checksum on a bunch of large files, and i think it's safe to say that two very large files can be assumed to be the same if the first 5mb, the middle 5mb and the overall size are exactly the same. Perhaps I will add an option later for strict checking, if someone is highly concerned about data integrity. But the practical limitations are there, I'm 30,000 files into a scan of my ~200GB backup folder and I certainly wouldn't have gotten that far without the file size limiting.

Update: The scan was almost done when I wrote this. Here is the tail end of the log:

31000  files scanned
31500  files scanned
32000  files scanned
32500  files scanned

real 43m40.531s
user 7m25.593s
sys 3m35.540s

So 233 GB over 32500 items took about 45 minutes to check and it looks like I've saved about 4 GB. Upon further inspection, it seems that most media files save their metadata in the file contents, so the checksum is different. Hmmm....

3.06.2010

File Management Tool - Part 2

0 comments

I have worked with my program a bit more and there are some interesting aspects of this kind of backup.

At the end of the backup all duplicate files point to the same inode, so the SHA1 version can be deleted.

eg. 

inode   name
1299    workingdir/folder1/file1.txt
1299    workingdir/folder5/file1.txt
1299    workingdir/folderx/file1_renamed.txt
1299    backupdir/54817fa363dc294bc03e4a70f51f5411f4a0e9a9

All these files now point at the same inode and so the backup directory can be erased and no file has executive control over this inode. All three files would have to be deleted to finally get rid of inode 1299. Generally it seems that programs save files with new inodes (Text Edit ...), so editing any of the versions breaks the links. It seems that UNIXy programs respect the inode better, vim saves with the same inode and so editing any version edits every version.

Removing the "backup" directory also helps Spotlight resolve the names and filetypes. Deleting that folder and running `mdimport ./workingdir` complained mightily but more or less re-indexed the folder. Here is a quick slice of the errors it produced, I'm not going to try to make sense of them, but think they're interesting; maybe Spotlight encounters these kinds of problems always and just keeps silent about them.

$mdimport ./workingdir
...
font `F88' not found in document.
font `F82' not found in document.
font `F88' not found in document.
font `F82' not found in document.
font `F88' not found in document.
font `F82' not found in document.
font `F88' not found in document.
encountered unexpected symbol `c'.
encountered unexpected symbol `c'.
encountered unexpected symbol `c'.
encountered unexpected symbol `c'.
encountered unexpected symbol `c'.
encountered unexpected symbol `c'.
encountered unexpected symbol `c'.
choked on input: `144.255.258'.
choked on input: `630.3.9'.
choked on input: `681.906458.747'.
choked on input: `680.335458.626'.
choked on input: `682.932458.507'.
choked on input: `530.3354382.624'.
font `Fw' not found in document.
font `Fw8' not found in document.
encountered unexpected symbol `w6.8'.
encountered unexpected symbol `w0.5'.
font `Fw8' not found in document.
encountered unexpected symbol `w0.5'.
encountered unexpected symbol `w6.8'.
choked on input: `397.67.'.
choked on input: `370.5g'.
choked on input: `370.5g'.
choked on input: `D42.32 m
314.94 742.32 l
S
306.06 751.2 m
306.06 7...'.
choked on input: `67.l'.
choked on input: `67.l4'
failed to find start of cross-reference table.
missing or invalid cross-reference trailer.

To reiterate, this is a funny trick that my program is doing. It builds a list of SHA1 named files from the source directory and then you just delete the index it just made and you're left with all the duplicates hard linked. I think that's pretty cool.

Metadata

One stated aim of this backup tool was to preserve metadata. So far this tool preserves the time stamps and metadata of whatever file it indexes first and the filename of every file it indexes. I'm not sure how to implement any more than this in a transparent way. As far as I can tell from the documentation, you can't have a single inode with multiple access and modification times. And building an external database of that kind of information would not get used.

File Management Tool

0 comments

I ended up sketching out the details of how my file managing tool will work, it's kind of like a virtual librarian that removes redundant files without deleting the file hierarchy. My method is a bit of a mashup of how other tools work, so I'll give credit where credit is due.

This is how it works so far:

  • All files in a tree have their SHA1 hash-value computed (Git)
  • A hard link is created in the backup folder with the SHA1 name (Time Machine) ...
  • unless: the file exists already, then it is hard-linked to the existing SHA1 (...)

There is no copying or moving of files, simply linking and unlinking, so 99.9999% of the time is spent computing the hash values of the files. Here's the python version of this:

import os
import hashlib

backupfolder = os.path.abspath('./backup')
srcfolder = os.path.abspath('./working')
srcfile = ''
backupfile = ''

for root, dirs, files in os.walk(srcfolder):
    for name in files:
        if name == '.DS_Store':
            continue
        srcfile = os.path.join(root, name)
        sha1 = hashlib.sha1(open(srcfile, 'r').read()).hexdigest()
        backupfile = os.path.join(backupfolder, sha1)
        if os.path.exists(backupfile):
            os.unlink(srcfile)
            os.link(backupfile, srcfile)
        else:
            os.link(srcfile, backupfile)
        # print backupfile

This folder contains about 5 GB of info and I thought that the SHA1 calculations might take a couple weeks, but as it turns out, it only takes a couple minutes. What you end up with is a backup folder that contains every unique file within this tree named by it's sha1 tag, and the source folder looks exactly as when you started, but every file is a hard link.

So, what are the benefits?

Filenames are not important

Because the SHA1 only calculates the contents of a file, filenames are not important. This is important in two ways, if a file has been renamed in one tree, yet remains physically the same, you only have one copy and the unique names are preserved. And more importantly, if you have two files in separate trees that are named the same, (ie. 'Picture 1.png'), you keep the naming, yet have different files.

If you have some trees of highly redundant data, this is the archive method for you. My test case was a folder of 15 direct copies of backup CD's that I have made over the years and I have saved about 600M across 5GB. And the original file hierarchies look exactly the same as they did before running the backup.

What is wrong with it?

As it stands, it messes with Spotlight and Finder's heads a little bit. Finder isn't computing correct size values for the two folders. du prints the same usage whether I include both folders or one at a time, which is pretty clever: total:5.1GB, working:5.1GB, backup:5.1GB. Finder on the other hand prints Total: 5.1GB, working: 5.1GB, backup: 4.22GB.

Spotlight

Some very wierd stuff happens with Spotlight.

A Spotlight search in the working directory will show mostly files from the backup directory, which isn't convenient. The files in the backup dir have no file extension so they're essentially unopenable by Finder. Here's what i found using the command-line mdfind:

mdfind -onlyin ./working "current"
/Users/.../backup/5f5b587eb07ee61f15ab0a032ca564a17ff461e9
/Users/.../backup/0f3f769000f164b2e30bb7b3f09482e8cc244135
and so on ...

mdfind -onlyin ./backup "current"
nothing found

For some reason, when searching the working directory it finds the information, yet always resolves the name of the file to a directory it's not supposed to be searching. And if you search the backup directory, it doesn't even bother reading the files, because it assumes from the name that they are unreadable by it.

I'm starting to wish that Steve Jobs hadn't caved and given in to the file extension system.

Time Machine

Okay, it's useful but how is it similar to Time Machine? Time Machine creates a full copy of the tree when it first backs up the system. From then on it creates the full hierarchy of directories but all the files that haven't changed are hard links to the original backup. Each unique file is a new inode created in time, whereas in my system each unique file is a new inode created in space. All duplicates in time are flattened by Time Machine and all duplicates in space are flattened by my system.

Note: To copy folders from the command line and preserve as much as possible for metadata use `cp -Rp`.

3.05.2010

File Backup And Synchronization

0 comments
In my previous post I had mentioned that I was looking for a backup/file synchronization tool.
I don't think Git is it and neither is dropbox. Both these are useful in that they are format transparent, which most database software is not. But what they are lacking is a way to deal with a large variety of file and folder hierarchies and seamlessly compress without losing transparency and semantic meaning.
So here is my list of requirements from a backup tool:
  1. Preserves any time-stamp information, even conflicting
  2. Distributed (decentralized)
  3. Minimizes redundant data
  4. Preserves hierarchies for semantic meaning
  5. Hides hierarchy clutter
  6. Preserves every bit of metadata, even if it's not explicit
  7. Accessible and platform neutral
  8. Makes data integrity paramount
It may seem like I have requirements that conflict with each other, but I will try to explain what I mean. I have loaded four of my backup CD's onto my laptop. I know there are duplicate files and I know there are time-stamps that disagree with one another.
I want to be able to view these files in a number of ways:
  • In their original on-disk hierarchy.
  • By file type, date, tags or physical description.
And I want to be able to synchronize all or part of these folders between machines, in addition to making zip/tar archive of them to a backup machine.
Any suggestions, or shall I start coding?

3.04.2010

Git And The Future Of The Internet.

2 comments
I've recently taken a detour into philosophizing about where technology is going. What does the future look like and what does it mean for humanity, life and the current business models as we know them?

It all started with a bit of research into Linus Torvalds latest project, Git. I've been thinking about trying some kind of content management system for personal use. I've looked at a lot of personal database type stuff (Bento, FileMaker, MySQL, ...) and they just seem like format specific black holes to drop your content into. I'm still not sure Git is right for what I'm thinking, but I watched Linus' Google tech talk followed by Kevin Kelly's TED talk and had a vision of a web that is so much more than what it is right now.

They're both pretty long, but I've had a bit of time on my hands lately... http://www.youtube.com/watch?v=4XpnKHJAok8 http://www.ted.com/talks/kevin_kelly_on_the_next_5_000_days_of_the_web.html Linus brings up two important points in his talk: one is the notion of working in a "network of trust" and the other is the sacredness of one's own data. Both of these are extremely important and often lacking components in the emerging technologies of our day. The network of trust is the only way to do collaborative work on open source development right now.

I think this is hitting a critical mass and will soon be the only way to do any kind of work. Monolithic organizations cannot keep up with the changing landscape of information growth. Git is a very interesting project because it takes this model and implements it in a very practical way. It employs a lot of very technical algorithms to allow software projects to grow very organically in a social environment. A lot of the metaphors that surround software development are hard, physical metaphors like construction, building and engineering, but the emerging metaphors are about growth, evolution and adaptation to environment. 

The benefits of collaborative networked projects are obvious but the sacredness of one's data is a bit more of a veiled concept. Linus outlines the use of the SHA1 algorithms as a means to ensure that the entire history of a project, or set of data, can be verified to be accurate and traceable throughout it's lifespan. This has obvious benefits when dealing with buggy network connections or failing hard drives, but it's more interesting to me in it's wider application.


Where's My Information?

As a person that has used a computer for a number of years I'm already seeing the breakdown of continuity in my archived information. As data gets moved around, archived to CDROM, uploaded to Google Docs, downloaded to PDF's and transferred to different operating systems, it all ends up in a soup of data without context or history. I have no idea if the timestamps are accurate, or what the context and related content might be. As soon as you add cloud computing to the mix, the problems amplify greatly.

This very blog post is being submitted to the vast expanse of content controlled and managed by the cloud. I have no simple way of traversing the internet and picking up all the odds and ends that I have put there.
This is the real direction of Git I think, and I want to figure out how to use it for more than just source code management because I think it could change the way the internet works. What if this blog was simply a mirror of the "Blog" folder on my hard drive, which was mirrored on every machine I use and was also shareable to other collaborators who mirrored their own unique versions? And what if my photo page on flickr and Facebook were simply mirrors of a folder called "Published Photos" on my hard drive which were mirrors of... and so on.


Vapor Trails

The fundamental problem of cloud computing is the owners right to content and tracking. This is generally possible with today's technology, but never practical. I have 65 documents in Google Docs at the moment and I could download all of them in one go into plain text files, but all the metadata would be garbage, and I couldn't easily merge them with the existing contents of my hard drive. Sure, I could spend a bit of time diff-ing them with my files and organizing them into logical places, but imagine if I was talking about the entire contents of my home directory. du | wc -l command shows 5,627 files in my home directory and I don't even have my music collection on this computer! Yes, the data is basically safe in the cloud, but what if I want to take it with me or move it elsewhere? What if I want to host this blog from my own server, how would I transfer it? The current cloud model only takes uploading and viewing seriously and neglects personal ownership rights. Google docs has special code written for exporting, blogger doesn't, facebook and flickr don't, youtube doesn't.

They are all greedy information gathering tools. They are only concerned with gathering your information and storing it on their sites. There are "sync" tools for most platforms, but their only intent is to gather your content with more ease and transparency.

Git looks promising in that it allows you to publish your information, yet still control the source of it.

3.03.2010

Assembly Language For Mac

0 comments
I'm away from my Linux box and want to do some assembly programming. Mac installs GCC with the developer tools, but there are enough differences that I haven't bothered to work through them until now. Here's a decent tutorial, although it focuses on PPC assembly and I'm using an Intel Mac. The thing that frightened me about the Mac assembler was the default output of gcc -S. There is some strange optimizations and flags in the resulting assembly code. The key, as the tutorial points out, is in the compiler options. Here's what I used on the ubiquitous "Hello World" program:
gcc -S -fno-PIC -O2 -Wall -o hello.s hello.c
And here's the assembly code it spit out:
    .cstring
LC0:
   .ascii "Hello World!%d\12\0"
   .text
   .align 4,0x90

.globl _main
_main:
   pushl   %ebp
   movl    %esp, %ebp
   subl    $24, %esp
   movl    $12, 4(%esp)
   movl    $LC0, (%esp)
   call    _printf
   xorl    %eax, %eax
   leave
   ret
   .subsections_via_symbols
This is more familiar territory, the only differences being the .cstring directive instead of .section .text, the leading underscore on printf, and the .subsections_via_symbols directive. The general naming of sections is outlined on the Mac Assembler Reference, and the .subsections_via_symbols explanation is interesting. I'm already used to using many labels in my code; does this mean that the named sections would be ripped out because they are not "called" by any other code? I tested this out in the previous example, just adding a second call to _printf in a labelled section and the code worked just fine. It seems that labels don't count, they have to be declared sections like .globl, .section or whatever. That seems fair, I haven't yet made a habit of calling sections that are supposed to flow naturally into other sections. Maybe there is some instance where this might be a useful optimization? I will be looking into Position Independent Code(PIC) a bit more, it seems that it's similar in theory to how the latest Linux kernel runs code at randomized memory locations to prevent hardcoded attacks, but I don't know if that's the extent of it.

3.02.2010

RPN Calculator - v0.02

0 comments
My calculator code was quite easily polished up. Here's the revised code which stacks operands properly and supports the main arithmetic operators +,-,*,/. If you flush the stack completely, you get a "nan" warning, which seems reasonable. Here's the code: (gas, x86)
.section .data
expr_length:    .int 128
ADD:            .ascii "+"
SUB:            .ascii "-"
MUL:           .ascii "*"
DIV:            .ascii "/"
null:           .ascii "\0"
disp_float:     .ascii "%f\n\n\0"
.section .bss
    .lcomm expr, 128
.section .text
.globl main

main:
    finit
    1:
    leal    null, %esi          #Clear the expr buffer
    leal    expr, %edi
    movl    expr_length, %ecx
    cld
    lodsb
    rep     stosb
    addl    $4, %esp
    pushl   stdin               # Read an expression
    pushl   $64
    pushl   $expr
    call    fgets
    addl    $12, %esp

    movb    ADD, %ah            # Test For Operators
    movb    expr, %bh
    cmp     %ah, %bh
    je      addFloat
    movb    SUB, %ah
    cmp     %ah, %bh
    je      subFloat
    movb    MUL, %ah
    cmp     %ah, %bh
    je      mulFloat
    movb    DIV, %ah
    cmp     %ah, %bh
    je      divFloat

    pushl   $expr               # Must be a number
    call    atof
    addl    $4, %esp
    jmp 1b

    addFloat:
        faddp
        fstl   (%esp)
        jmp     disp_answer

    subFloat:
        fsubrp
        fstl   (%esp)
        jmp     disp_answer

    mulFloat:
        fmulp
        fstl    (%esp)
        jmp     disp_answer

    divFloat:
        fdivrp
        fstl   (%esp)

    disp_answer:
        pushl   $disp_float
        call    printf
        addl    $8, %esp
        jmp     1b


    notfound:
    movl $1, %eax
    movl $0, %ebx
    int $0x80

Simple RPN Calculator

0 comments
Reverse Polish Notation is a simple method of calculation that was used extensively in scientific calculators such as the HP 32S but has fallen out of use somewhat these days. I have decided to implement an RPN calculator in Assembly Language to test what I have learned so far. Here is version 0.01 that only adds and subtracts, just to give a rough layout of how it will work.
.section .data
  expr_length:  .int 128
  ADD:          .ascii "+"
  SUB:          .ascii "-"
  null:         .ascii "\0"
  disp_float:   .ascii "%f\n\n\0"
.section .bss
  .lcomm expr, 128

.section .text
.globl main

main:
    finit
1:
    leal    null, %esi          # Clear the expr buffer
    leal    expr, %edi
    movl    expr_length, %ecx
    cld
    lodsb
    rep     stosb               #__
    addl    $4, %esp
    pushl   stdin               # char * fgets (char * str, int len, * stream)
    pushl   $64
    pushl   $expr
    call    fgets
    addl    $12, %esp

    movb    ADD, %ah            # Test For Operators in expr
    movb    expr, %bh           # and jump to code if operator found
    cmp     %ah, %bh
    je      addFloat
    movb    SUB, %ah
    cmp     %ah, %bh
    je      subFloat

    pushl   $expr               # Defaults to a number
    call    atof                # double atof (const char * str)
    addl    $4, %esp            # pushes a float into st(0) from string
    jmp 1b


addFloat:
    faddp
    fstpl   (%esp)
    pushl   $disp_float
    call    printf
    addl    $8, %esp
    jmp     1b

subFloat:
    fsubrp
    fstpl   (%esp)
    pushl   $disp_float
    call    printf
    addl    $8, %esp
    jmp     1b

    movl $1, %eax           # All roads jump back to 1 so we never get here
    movl $0, %ebx
    int $0x80
And here's how it looks in action:
$./rpn
14.5
88
+
102.500000

12.3
6.6
-
5.700000
It took some digging to figure out how to load a value into the FPU from a string and the "atof" C library function seemed to be the easiest. All it needs is a string pointer on the stack ("$expr" in the code) and it does it's best to convert it to a float and push it onto the FPU stack - fairly painless. The operator test needs some work since it will always give a false positive if you punch in a signed value. Punching in -12 causes the calculator to run the subtraction code and disregards the number you typed in, not ideal. Other todos: 1. Pull the display code out of the calculations, this should be generic. 2. Push the result back onto the stack so you can use it in following calculations. 3. Filter the input somewhat.

2.13.2010

px Language - Speed Tests

2 comments

In order to more fully understand how I'm going to implement the translator from my imaginary programming language, I have written a few direct translations from px to assembly. My thinking is that if the language doesn't rely on the abstractions provided by the "compile to C" interpreted languages, I should be able to end up with a systems level language with the clean syntax of python, ruby, et al.

My first example is to calculate the sum of a large list. Here are two versions in python.

UPDATE!!! After trying a few more versions, I realized that my loop counters were off in every example. I had made the silly python mistake of forgetting the range of range. range(0, 123456789) = 0,1,2...123456788. And I had subsequently propagated that mistake into my C and Assembly versions, producing the wrong answer in every case, sometimes very fast! Lesson learned: Check your math!

# listsum1.py - calculate 1 + 2 + ... + 12,345,678 = listsum

listsum = 0
for f in range(0,12345678):
    listsum = listsum + f
print "The result is " + str(listsum)

# listsum2.py
listsum = sum(range(0,12345678))
print "The result is " + str(listsum)

And here's what it might look like in px.

print listsum buildlist 12345678

But, of course, no library functions exist yet, so I'll spell out how the functions listsum and buildlist would be defined.

n... buildlist size, next 0:
    next next + 1
    n next if next < size
    :
n listsum list...:
    break if not list
    n n + list
    continue True
    :

buildlist is defined as taking one argument, size and producing a list, one item at a time. Listsum consumes these items and produces a final value when the whole list has been read. All items are passed as two 32-bit values, [type/status] and [value/pointer]. Each operation checks the status flag to check if it is receiving a valid object and what that object is exactly. The buildlist function uses this fact to push a false object onto the stack when it reaches "size". This tells listsum that there are no more values to be read. In assembly this is what the code structure looks like in simplified form:

list_size:  .int 12345678
l_sum:     .int    0

push $1
push list_size

pop list_size
pop edi            #using edi as the status flag
mov 0, ecx    #and ecx for the counter
buildlist:
    compare list_size, ecx
    cmovae 0, edi   #if above, set edi to 0 (false)
    add 1, ecx
    push edi
    push ecx
    listsum:
        pop eax
        pop edi
        cmp 0, edi
        je end_listsum
        add eax, l_sum
        jmp buildlist
    end_listsum
    
    print l_sum

In the actual implementation, I changed the add to support a quadword because the results get large quite fast, but this gives you the idea.

Is It Fast?

Well, I knew it would be fast, but this is a pretty large list, the code between buildlist and end_listsum has to run over 12 million times to calculate the sum, this is what I got when comparing the two python versions and my assembly version:

$ time python listsum1.py
The result is 76207876467003

real 0m15.379s
user 0m14.819s
sys 0m0.557s

$ time python listsum2.py
The result is 76207876467003

real 0m8.123s
user 0m7.476s
sys 0m0.647s

$ time ./listsum
The result is 76207876467003

real 0m0.275s
user 0m0.273s
sys 0m0.000s

That's pretty blazing fast if you ask me! But I was put in my place when I ran the C version which produced these results:

time ./f_sum3
The answer is 76207876467003.000000

real 0m0.123s
user 0m0.120s
sys 0m0.000s

I used a floating point double for the list sum, so maybe that's why C was faster. Still, if we use C as the baseline we have:

C       1
px      2.2
Py2     66.04
Py1     125.03

Not bad for a start. Now I'm off to dig through the compiled C code to see how it should be done...

UPDATE #2 - I've realized that there is a quite simple algorithm to solve this particular problem and that is: (n/2) * (n+1). This simple equation eliminates the loop completely making the calculation trivial. The updated python version runs in 0.038s and the C version runs in 0.002s producing the, now correct sum, 76207888812681.

2.11.2010

px Language - Parsing Challenges

0 comments

After spending most of the day investigating parsers, lexers and compiler compilers, I've come to the conclusion that my language doesn't really need these just yet. I've really only outlined three syntactic rules and two structural ones, and these cover all the ground I can see in front of me.

Syntax

1. Right to left evaluation, except inside parentheses:

z  y if (y > x)
z  x if (y 
2. Right to left definition of block header line:
z function z, y:
   z do_something y
   :
3. Comma seperated values are globbed together into an array.
process_list x, y, 12, 72, "Hello World!"

That's basically it for the syntax, where it gets interesting is namespaces and typing.

Structure

Every code block defines it's own namespace by default. This means that every block of code is explicit in defining what values it is using. Here's a trivial block which takes two operands and produces another:

.z a_function .z, y:
   .z y + .z

The ".", and the lack thereof, are the essential parts when it comes to namespaces. You might have seen the dot syntax in other languages (my_array.append()) and it's not really that different in px, except that referencing variables is a two way street. a_function is declaring it's own namespace under the main program and thus becomes main.a_function. That's fine, everything in the main namespace can access it directly by it's bare name and we can keep stacking them just like in other languages.

But the trick here is the preceding '.' on the variable name z and the lack of it on y. What's actually going on here? What happens is that the function is defined with concrete references to the z in the parent namespace and one reference to a y which does not exist. This becomes a unary function, a partially applied function and is never called until it is supplied with a y. When it does get called with a value, the z in the parent namespace gets updated accordingly. This function might be more appropriately called "increment_z", let's see it in action:

.z 10
increment_z  1
print z             >>11
increment_z  8
print z             >>19

A standard for loop imports all names, but it does so explicitly in it's function definition. "Hold on," you might ask, "it's function definition???" How does a for loop have a function definition? Don't other languages define this as built in to the language? The answer is yes, but in px I have decided to expose all the gory details of language design to the programmer, and only hardcode the bare essentials. This is roughly how the standard loops are implemented as blocks:

.* loop .*, _start, _end:
   break _end condition
   continue _start condition
   :

The _start and _end tags are handles for the final assembly language implementation. The * wildcard states that any and all of the parent namespace will be accessible. This block definition has no pre-determined outputs or effects and so it's considered a function template. If called, the break and continue expressions will only be included if their conditions are met. Here's how that might look:

loop:
   break if .x 

Loop doesn't actually control the looping, the statements within it do. We could have omitted the break statement and used the condition of the continue to control the loop. Notice again that this block is a fully defined namespace and all external references must be used accordingly.

To summarize namespaces, block = namespace = object = method = class = control mechanism = symbol. If a block has fully defined inputs and outputs, it is treated just like any other literal value in the flow.

Types

I've stated that there are two structural elements to my language: Namespaces and Types. Well, I kind of lied: there's only one types are also implemented functionally, but I haven't worked out how and how much of the checking can be done in the final assembly code. The general idea is this: values that aren't known at compile time are put on the heap and indexed with a hash table. Values that are know (by single literal assignment, or explicit typecasting) are put on the stack / data segment. Type statements are just like any unary operator:

x number 225.993
y int 235
z string "Please assign me to z!"

They go right next to the value being assigned as a check for the correctness of the data being passed to it.

2.10.2010

px Language

0 comments
In my studies with assembly language, a couple of things have occurred to me: 1. Assembly syntax is SIMPLE. 2. Instructions, registers, stack pointers, etc. are incredibly confusing. What i mean by the syntax being simple, is that it is completely linear. If the code jumps around it explicitly tells you where it's jumping and why. All lines can be understood as self-contained statements and the only structure to the program is the structure you give it. With this in mind, I have set out to design a new language based on this syntax. The language will be functional and type-safe like Haskell, but it won't have ANY built in syntax. I have begun to build a parser in Python which does the px to asm conversion. It's very simple at this stage, but should work. The language is read, for the most part, right to left with the leftmost term(s) being the terminal element. The only exception to the rule might be infix operators, because I'm not sure if many would take up a language if they had to learn reverse-polish notation. The Basics Here's a simple example of naming symbols and comparing them.
x, y  20, 15
z 42
z x if x > y
z y if y > z
print z
x, y and z are assigned the values 20, 15 and 42. y and x are pushed onto the stack and '>' pops them and compares them, putting a 0 or 1 on the stack. The if statement takes two operands, the x and the boolean result of the '>' operation. It then pushes onto the stack x and [0/1] (secret rule #1). Secret rule#1: Values are always stored in two parts, a value and a flag. If the flag is false the assignment doesn't happen. I have chose to omit the assignment operator, because all evaluations logically flow right to left. You will see the benefit of using infix operators if you reduce this operation to it's RPN form, with parens added for clarity: z (if ( x (> y x))) The > and other mathematical terms can be understood with some practice in RPN, but the if statement is not very intuitive at all without infix notation. Another curiosity that needs addressing is that since this is stack-based assignment, the first assignment "x,y 20,15" just adds the numbers 15 and 20 onto the stack and then pops them off making y=20 and x = 15. Not the preferred functionality! Functions Simple enough for basic expressions, how do we implement more complex functions? I've carried the same logic into function expressions. Functions always produce the left-hand value based on the right-hand input, but you add a code block below the statement line. This is how it looks in practice:
x 15
y 12
.x some_function .x, .y:
   .x sum(.x, .y)
   :
print x
I almost went with the Python-esque whitespace-significance, but have decided for now to close blocks with a final colon. The function's header defines it's inputs and outputs. The function takes two values, x and y, passes them through the function block and produces x. There are no return statements, because the output symbol is explicitly given in the header line. Okay, the syntax is simple enough, but what can you do with it, and what are the leading periods all about? That's where namespaces come into play... Namespaces Each block defines it's own namespace. The previous function declared inputs of .x and .y. These are concrete references to the parent namespace. The output was also a concrete reference, so the function was fully defined and executed to produce the new value of x. Here's an example that, although it looks very similar, is actually quite different:
x 15
y 12
m generic_function m, n:
  m sum(m, n)
  :
x generic_function x, y
print x
This is more like normal function you would use, because it defines the function with generic inputs and outputs and then you use them on specific values. The function is only executed when called with some specific values. This opens the door for partially applied functions and the like. eg:
m part_func .x, m:
   m sum(.x, m)
   :
y part_func y

m const_function .x, .y:
  print x, y
  m True
  :
res const_function

.x defined_out m, n:
   .x sum(m, n)
   :
defined_out 8 1024
print x
This is a simple but powerful way to implement a lot of the functionality of high level language with only one syntactic convention. One final note about namespaces is that the .* symbol references the entire parent namespace. This allows one to implement transparent code blocks like loops. Eg:
x 0
x_max 11
.* loop_block .*:
   continue .x < .x_max     print .x     .x .x + 1     loop True     : 
Because the input and output are fully defined, the code executes and can access any of the variables, functions etc. of the parent block via the dot syntax. The loop and continue functions control the flow of the block and can be used in any block. 'loop' is just a "goto $start if expression is true" statement and 'continue' is a "goto $end if expression is false" statement. This block first checks it's condition, breaking if it is false, does some more stuff and then loops back to the start unconditionally. Alternately, one might want to omit the 'continue' statement and do the condition check with the 'loop' statement. Every block will have a start and end tag that can be used in this manner. Objects Because blocks are self-contained namespaces, there is another possible use for them, classes and objects.
. some_object .:
   name "some_object"
   value 42
   :
This block didn't specify any inputs or outputs. It is thus a concrete object which can be used as follows:
print some_object.name
But maybe I'm getting ahead of myself! Stay tuned for more developments and please give suggestions on the name, I'm not sure how or why I came up with "px".

Twitter

Labels

Programming (21) Assembly (7) Productivity (5) Goals (3) Planning (3) Writing (3) cognition (1)

Followers

andyvanee.com

Files