Pretty printing JSON with Homebrew

You have an endpoint returning JSON. You could do

curl http://endpoint | pbcopy

and paste the text into TextMate, switch the filetype to JSON, and press Ctrl-Shift-H to have Textmate format the JSON. But that's way too many clicks.

'jsonpp' is an excellent command line utility for Terminal for pretty-printing JSON.

 brew install jsonpp

It's very useful combined with curl and pbcopy. It's the difference between a wall of JSON

curl http://endpoint

{"domain": {"domain": ".com", "endpoint": "http://e.esteelauder.na.us.dev.ncsa.elcdev.net/rpc/response.tmpl"}, "machine_name": "united-states", "name": "United States", "language": "en", "images": [{"group": "splash-background", "url": "/media/uploads/splash-background.jpg", "height": 976, "width": 768, "device": "iPad", "scale_factor": 1.0}, {"group": "sidenav-3mb-option1", "url": "/media/uploads/sidenav-3mb-option1.jpg",...

 and a much friendlier:

curl http://endpoint | jsonpp

 {
  "domain": {
    "domain": ".com",
    "endpoint": "http://e.xyz.foo.net/rpc/response.tmpl"
  },
  "machine_name": "united-states",
  "name": "United States",
  "language": "en",
  "images": [
    {
      "group": "splash-background",
      "url": "/media/uploads/splash-background.jpg",

Rare stock switcheroo

Decided I ought to rebalance my stocks. In particular, Whole Foods (WFM) was too large a holding. I sold a quarter or so of my total and bought Solar City (SCTY), Canadian railroad (CNI), Sun Edison another solar company (SUNE), the South American Costco (MELI), and Priceline (PCLN). Time will tell whether this was a good move. Those are all stocks I already own so I'm adding to my winners. Whole Foods hasn't done quite so well and its dividend isn't compelling enough to stick with it. My thought is that those others can offer faster growth. I still own three fourths of my WFM investment so I'm still a strong believer - the stores are continually packed and people like good food. 

Experimenting with ZFS Failures

NOTE: ZFS NAS blogging moved to http://www.zfsnas.com!

Lately I've been building a ZFS-based NAS machine using the excellent NAS4Free. I've been wanting to use ZFS for years, and got rather excited when Apple was planing to include it with OS X years ago. Sadly that effort fell through (over licensing terms I've heard), but other OS' have picked it up, notably FreeBSD. After an external drive died recently (fortunately in a dual-drive mirror, so no data was lost), I decided it was time to take the ZFS plunge. I took purchasing advice from DIY NAS 2015 Edition, going with Brian's choice of:

• Motherboard (an ASRock Rack C2550D4I)
• 16GB ECC RAM
• Case (Silverstone DS380B) - which turns out pleasantly more compact than I anticipated. 

I differed slightly:

• Power supply, since Brian had trouble fitting his - his is 125mm long and SFX-sized power units are normally 100mm long. I chose a Silverstone ST30SF, which has a lower wattage but I'm only planning on using six drives. It installs easily into the case.
• Along with WD Reds and Seagates, I'm mixing in a pair of HGST Deskstar NAS drives, which are very well reviewed.
• NAS distro: I tried FreeNAS but failed to get it to boot (! - probably user error), so switched to NAS4Free which worked first time, and has been rock-solid.

Items were purchased from Newegg.com, Amazon.com, Fry's, and Best Buy. This was partly to obtain disks from separate manufacturing batches, so that failures would be unlikely to occur simultaneously.

While waiting for all the drives to arrive, I built a 3-disk RAIDZ1 configuration to perform tests on. Each of the 3 drives has a capacity of 3TB. RAIDZ1 means one of the disks is used for redundancy; instead of 9TB of storage space, there's only 6TB. For that capacity loss we gain resilience to any one of the drives failing. If a drive were to fail, we could simply replace it and ZFS would continue like nothing had happened! Let's try some experiments and see how that works.

Working Configuration

These shots from NAS4Free show the three disks, configured into one Virtual Device (vdev), inside one pool; the pool has one dataset.

To begin with, the pool is 'ONLINE' and all three disks are working fine.

Unplugging cables

Let's simulate a drive going bad; what happens if we unplug the SATA cable from one of the drives?

The pool is now DEGRADED and one of the drives is marked UNAVAIL. Uh oh. ZFS tells us to use zpool online to bring the drive back. I'm still able to access my data - in fact our home folder (~) is located on this dataset, and operating perfectly. Even with only 2 out of 3 drives running. (If we were to lose a second drive, our data would be inaccessible).
Let's re-attach the 3rd drive's SATA cable and tell ZFS to online the drive.

It all works great; I wonder why it says it "resilvered 68K". There are several MB of data in the pool. 68K is perhaps just some metadata.

Moving cables

How about we unplug a SATA cable from the motherboard and connect it to a different SATA port?

ZFS didn't blink an eye; we didn't even have to online the drive.

Let's pretend the entire motherboard needed replacing, and we forgot which drives were plugged in where. I shutdown; unplug all the SATA connectors and re-connect them in different connectors. Power on and what happens?

Switching connections around made no difference at all! If that had been a real motherboard replacement I would have had not had to worry where the drives were connected.

New boot drive

I've been booting this box off a 16MB USB flash drive formatted with NAS4Free 9.3.0.2.1310. What if this drive went bad? I installed NAS4Free on an 8MB flash drive. I told it to configure the network card (option 2 on the main menu) and then visited the displayed IP address from my Mac's browser.
Uh oh. Nothing here!

Here's the configuration screen, with no pool but a useful button labeled "Import on-disk ZFS config". Later we can see what ZFS commands the button runs.

After clicking that - look what happened!

The ZFS pool, vdev, and dataset are back! Oddly though, the Pools and Datasets web pages show nothing. (We can fix that, too - read on.)
At this point I enabled SSH access so I could have a poke around and try to access some data. I enabled root SSH access, and was able to navigate to the ZFS dataset directory!


zpool status shows the pool as ONLINE. Interestingly, zfs history shows all the commands that were used to create the pool and also what command was executed when we imported after booting from the fresh USB drive:


Though NAS4Free's WebGUI showed no pools or datasets, I was able to fix that using the ZFS -> Configuration page, which has a "Synchronize" button. After using that the rest of the WebGUI shows the pool and dataset correctly. This also fixed the Disks -> Management page, which had been showing no disks. As far as I can tell, that puts everything back the way it was (as far as ZFS goes - you may have had SAMBA shares etc too; so backup your NAS4Free configuration!)

Summary

I simulated loss of a hard drive, loss of motherboard, and loss of boot USB drive. These simulations of failures turned out to all be recoverable situations! No data was lost at any step, which is great news for anyone with data they want to keep safe.

Note that even ZFS is not a substitute for backups - preferably off site, e.g. at a family member's house or in a bank vault. An errant script or accidental manual file deletion means that ZFS will safely replicate that deletion across its RAID. ZFS snapshots could help here, but even so, your box is still vulnerable to flood, a lightning strike, power surge, or brown out which could damage one or all of the hardware components.
So far though I'm very pleased that ZFS, FreeBSD, and NAS4Free have lived up to their claims and provided a safe haven for my data!

I'll be adding three more drives and setting up RAIDZ2. This will allow data access even with two drives gone. Research has shown that RAIDZ1 is not as safe an option as you might think - once one drive goes bad the odds of a second following it shoot up, and may not give you time to resilver a replacement disk.


References:

http://www.nas4free.org


General overview and advice for ZFS, Freenas, and configuration: https://forums.freenas.org/index.php?threads/slideshow-explaining-vdev-zpool-zil-and-l2arc-for-noobs.7775/


NOTE: ZFS NAS blogging moved to http://www.zfsnas.com!

Commodore 64 Mandelbrot, in color!

Thanks to a great post by Semi-Original Thought, I witnessed a Mandelbrot set being rendered on a C64. I hadn't seen this since the 1980s! Back then the fractal inspiration was due to Scientific American magazine (Martin Gardner?), and the school library. 

Rendering is painfully slow on a real C64, but fortunately the VICE emulator has a Turbo mode; far faster than the original by about 50x.

Two programs were supplied by the Semi-Original Thought post; a lo-res rendering, and another for a 2-color hi-res render. The C64 supports more colors, so I figured I'd drop from the hi-res of 320x200 down to 160x200, which enables multicolor mode. For simplicity, I stuck with 4 colors; with fewer pixels to compute, the render is faster (still a few minutes even in Turbo mode!) 

Drawing very heavily from the Semi-Original Thought code, here's the multi-color version. The main changes are:

• The POKEing in the graphics section (lines 600-), to handle multi-color. 
• I added some machine code to speed screen erasing. It takes quite some time for C64 BASIC to clear 8000 bytes. In machine code, it's almost instantaneous. Lines 800- for that clearing code.
• Since the Mandelbrot set is mirrored about the x-axis, I skip computing the mirrored bottom half and draw 2 pixels for every computation. This also affected lines 600-.

  100 xl =-2.000:xu =0.500

  110 yl =-1.100:yu=1.100

  115 reps =20

  120 width =160:height =200

  130 xinc =(xu -xl)/width

  140 yinc =(yu-yl)/height

  200 rem main routine

  205 print "{clr}"

  207 gosub 500

  210 for j = 0 to int((height - 1)/2)

  220 for i = 0 to width -1

  230 gosub 300

  240 gosub 600

  260 next

  270 next

  290 get a$: if a$ = ""then 290

  299 goto 700

  300 rem calculate mandelbrot

  312 nreal = xl + i * xinc

  313 nimg = yl+j*yinc

  315 rz=0:iz=0

  316 r2z=0:i2z=0

  320 for k = 1 to reps

  330 r2z=rz*rz-iz*iz

  340 i2z=2*rz*iz

  350 rz=r2z+nreal

  360 iz=i2z+nimg

  370 if(rz*rz+iz*iz)>4 goto395

  390 next

  395 if k<7 then c=3:return:rem here we are assigning colors

  396 if k<9 then c=2:return

  397 if k<14then c=1:return

  398 c=0

  400 return

  500 rem clear screen

  530 poke 53272, 29:poke 53265, 59

  531 poke53270,peek(53270)or16

  532 poke 53281, 0

  541 gosub 800

  549 rem set screen colors to grays

  550 for r=1024 to 2023:poke r,207:next

  560 for r=55296to56295:poke r,14:next

  598 return

  599 rem draw hi-res

  600 ch=int(i/4)*8

  605 k=height-1-j : rem here we do the mirroring

  610 p =8192 + int(j/8)*320 + ch+(jand7)

  615 p2=8192 + int(k/8)*320 + ch+(kand7)

  625 lb=2^(7-(1+2*(iand3))):rem multicolor low bit

  626 hb=2^(7-(2*(iand3))): rem high bit

  630 if c=0 then return

  635 a=peek(p)

  640 if c=1 then pokep,a or lb

  641 if c=1 then pokep2,a or lb:return

  650 if c=2 then pokep,a or hb

  651 if c=2 then pokep2,a or hb:return

  655 a=a or lb

  660 pokep,a

  661 pokep2,a

  670 pokep, a or hb

  671 pokep2, a or hb

  680 return

  700 rem return to normal

  705 poke 53280, 14:poke 53281,6

  740 poke 53272,21:poke 53265,27

  750 poke 53270,peek(53270)and239

  760 end

  799 rem some machine code to erase the hires screen

  800 for i= 0to 20:read a:poke 49152+i, a:next

  810 data 169,0,162,32,160,0,132,33,134,34,145,33,200,208,251,232,224,64,208,244

  815 data 96

  820 sys49152

  830 return