Amazingly accurate comic from xkcd is amazingly accurate:

Some day I'll blog about the time a lib maintainer deciding to [literally] say [well, "post"] "Ho hum" about just such a dependency temporarily crashed a reasonably large company's deployment script, but that day is not today.

(Luckily it was reasonably easy to fix, but it certainly cemented my usual claim that, "If you adopt a third-party dependency, you should also be routinely submitting pull requests to it.")

Labels: dependencies, development, node

Had a strangely hard time finding good instructions on how to publish a Node project to the npm registry.

Ultimately found this blog post, which seems to have gotten me over the line.

To create an account, go to the npm signup page and fill up the form. After finishing the process, a test can be done using npm login and npm whoami. Has it output your username? Great! It is working!

A number of tutorials were missing that part as well as that you need (?) to use npm publish --access=public the first time you publish. (You can apparently use npm publish by itself afterwards.)

Here are some useful pointers on setting minimum Node version and how to version your app.

Labels: node, npm

I've been setting up a new greenfield project, and wanted to use not exactly a new stack, per se, so much as an "all-star" stack that'd capture the lessons I've learned coding recently. For me, that means it needs to be pretty straightforward, minimal dependencies, and minimal compilation. The fewer libraries and snowflake server requirements, the better.

On first take, after bypassing Node, which really is the winner,¹ my guess was that my all-star stack would be...

1. ASP.NET Core
   • C# with no OS limits.
   • Deploy inexpensively to Linode, eg, if you want.
2. Web API on ASP.NET
   • No Razor templates (not that I hate Razor; just one less tech to maintain)
   • No direct interaction between server and views/html/client.
   • Keeps concerns separated perfectly, exchanging only JSON between tiers.
3. Statically served html (with JavaScript packages, natch)
4. jQuery 3.x slim
   • Slim means no ajax; it's more efficient to roll your own.
   • Every time I've used $.ajax, we've wrapped that with our own error handling, etc, anyhow.
   • Learn to spell XmlHttpRequest. ;^)
5. Bootstrap
   • The cheapest mobile client is a responsive web client.
   • I realize native is better. But responsive is, give or take, free.
   • Also gives a decent basic theme.
6. VueJS
   • This was probably the most difficult to decide on ahead of time
   • Will discuss more below.
7. PostgreSQL or MySQL

VueJS was difficult to pick. I've used React, and almost solely React, for the last year. I like it. But I believe it contributes to monolithic source that's not necessarily factored well. The more interdependencies your code has, the more maintenance is an issue. And doing good React essentially requires, at this point, using Babel, and as soon as you're transpiling, well, the surrounding tooling and library dependencies skyrocket. Not to mention that debugging transpiled code either means you're wading into transpiler-generated muck, or you've got to maintain yet more tooling to step through things in something that's not the browser.

I was originally going to use Handlebars, which I've liked for years for its exceptional thinness & the way it encourages minimal business logic on the client, but caught Shawn Wildermuth on Jesse Liberty's Yet Another Podcast, and heard him talk about VueJS. VueJS is essentially Handlebars with, afaict, a few more recent lessons baked in. The syntax is actually very similar to Handlebars in many ways.

Here's what eventually won me over:

Unlike other monolithic frameworks, Vue is designed from the ground up to be incrementally adoptable. The core library is focused on the view layer only, and is easy to pick up and integrate with other libraries or existing projects. On the other hand, Vue is also perfectly capable of powering sophisticated Single-Page Applications when used in combination with modern tooling and supporting libraries.

It looks like we've got good room to grow using VueJS' conventions if we want to do a full SPA in the future.

I also debated not using Bootstrap, but I don't think it's worth the time to roll your own responsiveness when Bootstrap does it so well. It's over 200 KB total, but if you have it cached, that's a one-time hit that's smaller than most images. You're fine.

jQuery was also debatable, since you can handle DOM manipulation fairly well by wrapping vanilla JS with some convenience wrappers to check for DOM existence, but at 69 KB? Come on. It's well known, rock solid, and has good industry support. Use jQuery.

This looked nice. Simple, API-only and REST convention compliant server. No licensing fees. No JavaScript transpilation required, which will pay for itself in spades when you're debugging. Thin client. Clean separation of concerns.

I was going to go through my headaches setting everything up here, but I'll split that into a new post later. The quick version? Because of a wack nuget configuration I didn't realize was an issue (command line nugget didn't pick the "right" nugget server; I had a local one set up in VS2017 as well), I'm using SQL Server for now, though I will probably give PostgreSQL another look before I settle on a dbms.

Labels: c#, javascript, node, noteToSelf, web

JavaScript transpilation is a neat trick, but in practice you may seem to simply trade one multifaceted, shifting platform -- what works crossbrowser -- for another -- here, a set of constantly changing hipster libraries.

There are simple solutions to the shifting , each with one major flaw:

• Target a "lowest common denominator" or "limiting reagent" browser (ie, IE[x]) with "native" JavaScript.
  • This stinks because even IE11's tools are much worse than Chrome's DevTools.
  • Yet if you don't live in your "worst" browser, you're going to miss issues. No, you are. Every time.
• Pick one set of transpiled technologies & practices, and, within reason, stop.
  • You're still going to have to test in browsers
  • You'll still be responsible for debugging third-party code.
  • You'll disappoint your hipster programmers.

Interpreted incorrectly, transpilation gives the impression that you're developing for an environment like a server: a controlled, homogenous system. You're not. It's not node. And I've yet to see a practical advantage beyond very specific, situated uses -- JSX compilation for React, for example -- provided transpilation. It's all still very familiar zeroes and ones.

Labels: crockford, javascript, node, transpilation

At some point I'm going to write up all the crap I've done to host node.js on Linode easily. It's really kind of a pain if you're coming into it fairly cold. I mean, I've been using *NIX since the 90s, but when you really are the only schmoe responsible for everything, it's a new experience.

There are three main things I needed to learn to code and release node:

1. How to set up firewalls -- ufw and iptables.
2. How to execute node in an extended test mode that didn't shut down when I logged out.
   • Both for testing and for production
3. How to fix case sensitivity
   • Okay, I'm a dope. As you may remember, this one got me, though I have better fix now.

I'm going to hit number two right now, just for fun.

Use forever

Here's a good resource, but let's hit the high points:

$ [sudo] npm install forever
$ forever start simple-server.js
$ forever list
    [0] simple-server.js [ 24611, 24596 ]
$ forever stop 0

See how the forever stop 0 matches the [0] from forever list? Then you've got it.

... and one not on the list...

$ forever stopall

Success.

Why is that useful? Two reasons. First, I have a testing service set up that mirrors the "real" site so that I can stage changes both to my website and to the express server. That's at another port, and it's good to be able to start that code from the command line with something better than node app.js (or nodejs app.js as the case may be. If things act funky, try a which node and start sniffing).

But I might want that staging server to be up indefinitely. I probably don't really need it to stay up if the VM blows up, but I want it to stay even when I move from tower to laptop and back. Forever does this nicely.

Secondly, to "really" keep things running, you want to create something for systemd, which apparently replaced upstart in Ubuntu to some controversy a while back, that also ends up calling forever.

Fastest intro to systemd EVAH

$ cd /etc/systemd/system/
$ [sudo?] vim myservice.service

Now put something in the service file like this:

[Service]
ExecStart=/usr/local/bin/forever /home/yourLogin/path/to/your/app.js
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=node-sample
#User=srv-node-sample
#Group=srv-node-sample
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target

Now [esc]:wq (for you vi[m] users).

That seems to be alls you need. Notice that the ExecStart part has forever (and the full path to forever in it).

Here's the rest of the magic:

$ sudo systemctl start myservice.service
$ sudo systemctl status myservice.service
● myservice.service
   Loaded: loaded (/etc/systemd/system/myservice.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2016-11-30 12:16:47 EST; 1 weeks 5 days ago
   Main PID: 30575 (node)
      CGroup: /system.slice/exweb.service
              ├─12345 node /usr/local/bin/forever /home/yourLogin/path/to/your/app.js
              └─12346 /usr/bin/nodejs /home/yourLogin/path/to/your/app.js

$ sudo systemctl stop myservice.service

And now you're booting up your node server indefinitely or as a service in case of reboot.

I'll try to remember to follow-up on the other two. The case sensitivity one is the more interesting problem; it's borderline fun. ;^)

Labels: indie, linode, node, noteToSelf

From Tips for Writing Portable Node.js Code:

Bonus: Something that Breaks on Linux and Solaris!

Both Windows and, by default, OS X, use case-insensitive file systems. That means if you install a package named foo, any of require("foo") or require("FOO") or require("fOo") will work—on Windows and OS X. But then when you go to deploy your code, out of your development environment and into your Linux or Solaris production system, the latter two will not work! So it's a little thing, but make sure you always get your module and package name casing right. [emphasis mine -- mfn]

This is a problem when serving files by name with express' static middleware too. Argh. And check out this schmoe's problem, with a huge legacy php site migrating to Linux, from case insensitivity to case sensitive. Ouch. Can you imagine the tech debt there?

So what if you want to use Express to serve files from the filesystem on, I don't know, Ubuntu on linode [which uses ext4]? And what if you want folks to be able to access your privacy policy at both http://rufwork.com/Privacy/ and http://rufwork.com/privacy/?

Even worse, what if you told your app reviewer to try http://rufwork.com/privacy/ and that meant they had to reject your app submission now that you've changed servers? /facepalm

From MarkUpDown - Review Results:

Locations: mumduwp1.4.0.0x86x64_arm.appxbundle

Tested OSes: Windows.Universal

Tested Devices: Acer Iconia W700

Notes To Developer

The privacy policy link did not resolve to a functional webpage.

(>ლ)

I've never liked it when I've typed in a URL by hand and got a 404 because I had one or two letters in the wrong case. I mean, it makes sense that case sensitive file systems are easier to maintain. It's just an end-use-case pain when you're serving files.

But URL case in Edge is broken too... 🙄😱

And insult to injury? Microsoft Edge (Internet Explorer's replacement in Win10) "remembers" case when you retype a URL! What, wait? So I can't navigate to the "correctly cased" URL with Edge even if I want to -- and I wonder if I get the same reviewer(s) if this'll happen to them too.

(>ლ)

How to fix it

How do you fix case sensitivity for express so that it's never a problem with ext4? Well, you could make everything in your filesystem lowercase, and then just always run request.url through .toLowerCase(). That seems pretty nuclear. Slightly better is to try the URL as requested first, and then do this...

From expressjs.com:

When a file is not found, instead of sending a 404 response, this module will instead call next() to move on to the next middleware, allowing for stacking and fall-backs.

Still have to set the filesystem to a single case throughout, and then if you want to use mixed case in your URLs to make them easier to remember or what-have-you, you're creating a good deal of extra work for your server every time mixed case is used.

How bad is it to make your web server take a URL, not find it, then search the filesystem again for a lower cased version? It might be smarter/less work just to lowercase the string every time and ensure your filesystem is in sync. It has to be cheaper to lowercase the url every time than to hit express.static twice in 30+% of your requests, right? Kind of a pain when you're editing files and want to glance at human readable names, but I think that's the only other downside.

So I guess it looks like I get to go through and change the case of my website's files and folders to a single case. Right? Isn't that The Right Thing to do to approximate case insensitivity? Is it ever better for the end user to have a case sensitive http server? I don't think so. Seems unnecessarily processor churny and hacky to lower case the earth, but man, I hate case sensitivity for URLs.

That fishing metaphor that tells you I'm about to comprise my best intentions...

In any event, for now, I've decided to embarrassingly give the server a fish, I'm afraid.

if (req.originalUrl === "/privacy" || req.originalUrl === "/privacy/") {
    req.url = "/Privacy/";
}

That's probably worth a Unicode facepalm too.

I really enjoy using linode, though my needs are so small I might drop down to Digital Ocean's $5 a month plan if it's just as easy to administer. It's great having your own "box" on the net. So much more powerful than shared hosting. Feel like I was about a decade behind with hosts, but, boy, I'm caught up now. Wonder how long before I start using .NET Core on it...

Labels: case, coding, encoding, node, noteToSelf, rufwork

Turns out gzipping and writing the now gzipped buffer to a file in node is pretty easy. Not sure why this took me so long to put together. Guess I'm still getting used to node's buffers and streams. And I didn't bump into something that showed how to gzip and write a buffer that's already in-hand to a file in node quickly either. Strange. Everyone uses ExpressJS, afaict.

I wanted the crap web server I'm writing in node to be able to deliver gzipped content, and thought the neatest way to do this was to check if a gzipped copy existed already for a requested file (of the right types -- html, js, and css). If not, I deliver the raw/uncompressed version initially and asynchronously fire off a request to start the compression for next time. There's also, obviously, some logic to see if the original is newer than the gzipped version, etc etc. I'll skip all that for now as I straighten it out, but will probably push to npm in a week or two. I really am [currently] worried all the overhead for each request (parse file paths, get stats on two files with protected/try blocks, and compare modified dates if they both exist) is going to kill much of the advantage cached & compressed files provide. Should test with some significant load, I guess.

But the actual gzipping isn't bad if you use the zlib gzip convenience method. Simplest case, hard-coded file name, version follows.

/*jslint unparam:true */
var fs = require('fs');
var zlib = require('zlib');

try
{
    var strPath = "C:\\temp\\temp.js",
    gzPath = strPath + ".gz",
    inBuf;

    inBuf = fs.readFileSync(strPath);

    zlib.gzip(inBuf, function (err, buf) {
        if (!err)
        {
            fs.writeFile(gzPath, buf, function(err) {
                if (err) {
                console.log(err);
            }
            else
            {
                console.log("Gzip successful: " + gzPath);
            }
            });
        }
        else
        {
            console.log("Error in gzipping: " + err);
        }
    });
}
catch (e)
{
    console.log("Error: " + e);
}

The reason I'm using a buffer if because I've already read and returned the raw version of the requested file, uncompressed, to the most recent requestor. There's no reason to make them wait until the cached copy is ready. They get the original now. But then I've got that buffer sitting around, and there's no reason to read the file twice...

(The reason I'm not using ExpressJS to serve static content is that I'm always wary about code you haven't vetted, and this didn't seem like a horrible task when I started. I'm playing back and forth about writing a dependency-less-ish version of this server, and then later adding a version that allows 3rd party dependencies that could use Express (etc) instead of the hand rolled stuff if it's installed. But there's so much overhead in Express... Find where it looks up content types, for instance. You're going to have to travel through three or four dependencies until you end up at the source. And it's not tuned for lookups, I don't believe. Not that it's a huge deal, but Express's minimalist claim? Thhhbth. I mean, I'm sure I'll figure out I've bitten off too much reasonably soon, but right now a focused server targeting delivery of single-page apps (static + tons o' JSON) seems like a doable smart idea.

Anyhow, I didn't take long before I was serving up "routings" (where the server parses the URL to see if maps to a registered function) and static files when no routing rule matched (if the static files existed). I'm not sure why I let myself get distracted by gzipping, other than it seems you oughta have it if you want to pretend you have a web server. /sigh)

Labels: expressjs, javascript, node

So though it's completely unfair to characterize a technology stack based on the poor implementations built on top of it by 3rd parties, I completely get what this guy is saying:

If you've spent as much time as I have plumbing the depths of .Net, you know as well as I that it's full of mean surprises that get marked as "won't fix". The vendor selling it to you doesn't care - they don't build actual stuff with it; it's just there to drive server license sales.

He's comparing that to using packages on Node. Here's his Node sum:

[Node is] one of the very few languages that has a package system that isn't a steaming pile of [nastiness] and the core libraries are small and useful. Most of the libs I've used are maintainted [sic] by decent folks who actually acknowledge issues and accept PRs.

I just started fishing around in node search, and it isn't all pretty. And it's not like .NET doesn't have any well-maintained, friendly, open-source projects, but I get his point here too. I'd probably even agree that as you move from .NET's core to its "core libraries", quality suffers. LINQ is great, but that's a core language feature at this point, I believe.

The nice part about Node is that it's very Linux-on-the-desktop-y, in that simply being a Linux user means you're willing to accept many things your standard workstation user would not.

Node usage presupposes a few things whose importance we might underestimate:

*Users are familiar with the command line. *They know JavaScript well. * They don't mind basing their livelihood on an open source library. * Every node app, at least the node part, is headless/UI-less.

That's a pretty select group of folks, and one I'd rather work with than the guys who need point-and-click admin interfaces (not that everyone who uses MS does, but those who do need the hand-holding are largely welcome there [1]) who think SSRS is the way to create web interfaces for their reports (and sympathy to anyone whose job forces them to use SSRS. You know, node and MS SQL really aren't that bad together...).

[1] I had MS SQL Server training years ago, and wow. The guy I was working with did almost everything from the command line, it seemed, or at least could, and would if it was easier than the GUIs, so that's how I was learning to do it too. But man, there were tons of people in the course who only knew how to run a SELECT by right-clicking a table in what's now SQL Management Studio and selecting the SELECT options from the context-menus there.

Last month, I took a training course on administrating VMware's vCOps/vROps. Same deal. Though you could use PowerCLI to pull out all of these metrics and then pretty much push them wherever you wanted, the course was all about how to left & right-click your way through wizards with exceptionally klunky UIs to make management-friendly "dashboards" whose metrics those admins might or might not actually understand.

Labels: node