Jim Carnicelli - Portfolio

After over a decade of web work I had settled into a strategy for larger web applications oriented toward rapid development and high performance. All of my implementations have used a thin ASP.NET middle tier that contains almost no business logic. Instead almost all business logic has resided in stored procedures in SQL Server alongside application data. Almost all UI code then resides in static JavaScript and CSS files downloaded to the client. The middle tier only exists as a sort of pass-through that glues the browser client to the database. Browser-accessible SPs all have a special prefix and rely on common procedures to limit access and otherwise prevent hacking.

Because the middle tier is really a thin "glue layer" I reasoned that it could be implemented using any significant combination of programming language, operating system, and web server. To that end I decided to try crafting a pure Node.js (not Express) middle tier version with all the hallmarks of my ASP.NET implementations. At first I had it connecting to a Sql Server database. Then I modified it to connect to a MySQL server.

In the same vein I reproduced the essential framework components in MySQL that I typically include in Sql Server. When a page request is received one central stored procedure is called that finds the appropriate page SP and metadata about it needed to craft a call to it. In Sql Server implementations that first SP constructs a dynamic SQL script and executes it directly. MySQL imposes stiff constraints on dynamic SQL. So I decided to let that SP return its findings and have the Node.js script make that second SP call.

I honed this application design pattern over about a five year period for one client. I was able to reproduce most of it in about a day using Node.js and MySQL.

Like many medical software systems, Triton, a medical transcription system, relies heavily on patient "demographics". When a doctor records an audio file representing a report about a patient, that patient has already been checked into the facility and an "encounter" record created for the visit. Amphion has the facility send us that patient/encounter/order information sent to Triton using the nearly ubiquitous HL7 standard format. The physician typically keys into the phone or other dictation system a medical record number, date of birth, encounter ID, etc. that roughly uniquely identifies the patient or encounter. The goal is that when the transcriptionist starts her work translating the audio into text, she'll already have the patient demographics associated with that dictation record so she doesn't have to enter that information.

An HL7 message can represent many kinds of events that can occur in a medical system. Its structure is roughly a 5 level hierarchic flat file. It may contain key information about the related patient (e.g., name and date of birth), the encounter (e.g., visit ID and discharge date), the order (e.g., details of a CT scan requested), and so on.

The service I created is responsible for creating new records and merging in changes to existing ones. Although the HL7 standard does provide a lot of consistency, most systems that support it have wide variation in how they choose to represent data. Further, facilities can customize HL7 mappings. The result is that each facility requires a unique set of mappings into Triton's demographics database. The system I designed allows for a shared set of mappings that all facilities inherit and the ability for administrative users to define custom overrides by facility. Further, each mapping can call out to custom scalar value functions (SVFs) in the database to do additional formatting and lookups as needed. As a result, nearly 100% of the mappings, including custom code, are represented purely in the database for maximal transparency and self-documentation.

I was responsible for developing the entire service and a series of Intranet based utilities for letting administrators edit HL7 mappings, search for historical messages, view HL7 messages in parsed detail, simulate import of any HL7 message, and reprocess any import.

Uploading files through the web is not new, but there are many approaches to the question of how to store and retrieve such files. One simple way is to store uploaded files in a folder somewhere under a web site's root folder (e.g., "/MyWebApp/Attachments"). One problem with this is that it is insecure by default. Anyone who knows the name of a file can retrieve it. Storing the binary contents of a file in a database solves this problem because the web app can serve as a gateway, controlling who sees which file.

Another problem with a public web folder is that it doesn't by default work with a web server cluster. One has to get or create a special service to copy every uploaded file to every other web server in the cluster or choose to store files on a dedicated "attachment server", which itself doesn't have a proper backup. Storing binary file content in a database solves this problem by making the same file contents available to all web servers in a cluster.

Storing files in Sql Server comes with several risks. For one, it tends to bloat database files quickly. A single JPEG may contain more data in it than hundreds of database records otherwise. This can become a problem with large databases. FileDump solves this by allocating a dedicated FileDump database catalog that is separate from the multiple catalogs that are used with the various apps that used FileDump.

Another risk is that it can be complex to deal with all the idiosyncrasies of transporting binary file content between a web interface and a database. FileDump solves this by encapsulating all of that in simplistic save and view methods. One simple example is that FileDump looks at a file's extension when the code wants to push the contents back to a calling web browser and decides which of hundreds of MIME file types to pass along, a key clue for the browser to decide how to handle the contents. Another example is the streaming model, which relies on a small buffer size to give users the ability to consume content even before it's fully downloaded, while minimizing the amount of memory used by the database or web server, even with gigabyte-size files.

In anticipation of the need for versioning of uploaded files, FileDump makes it easy for application code to choose which of 4 major version models (append, hide, overwrite, delete) to use with newer versions of uploaded files.

Although the code component was made with ASP.NET apps in mind, I made a point of putting most of the business logic in Sql Server stored procedures. This makes it easier in the future to develop mirror versions of the code component in other languages.

I also documented the entire object model.

Most of the better off-the-shelf charting components cost a few hundred dollars per machine they are installed on, but our scalability model called for putting clients on groups of isolated application clusters, each of which had multiple web servers. This would have meant spending many thousands of dollars up front and ongoing as we brought new clusters online. Creating our own component would save P1S money up front and in the long run.

In addition, most of the better charting components were designed to directly connect to Sql Server to fetch data, which did not fit our goal to funnel all database requests through apps' middle tiers. They also often feature XML interfaces, which also didn't fit our model very well. I kept WebCharts' interface very object oriented and simple, especially with respect to getting data into it. Our apps tended to favor explicit loops for data manipulation, so WebCharts' model suited that better than the alternatives.

One key requirement was the ability to mix bar and line charts and have two Y axes. Most off the shelf components didn't support this easily. WebCharts made it pretty trivial. WebCharts also made it very easy to explicitly control the axis, right down to the individual tick marks. One also had very fine control over the fonts and text for everything, from headers and legend entries right down to individual tick marks. One could, for example, set a single tick mark's label's font to be red and bold to stand out. This degree of flexibility even extended to the line and bar series and even individual data points, with the ability to customize the shape, size, color, and line style, transparency, etc. of every last point, line, and bar,

One failing of many commercial components is that they don't choose "smart" ranges. If the Y range for a line chart is 0.5 - 23.7, for instance, they usually will simply evenly divide that range into perhaps 10 lines at oddball intervals, whereas WebCharts by default always chooses ranges that are on clean orders of magnitude (0.1, 1, 10, 100, etc.), making the results easier for people to understand.

I also made a point of providing full documentation of the component and its entire object model. In addition to applying it to the Quantifi project I was working on, I also provided mentoring to developers applying it to other projects.

Quantifi's Uis seemed deceptively simple at first, a reflection of a basically good design that hides a lot of complexity. Most data would be entered in a main "document editor" web page. But one key requirement would allow our customers to customize the document designs, which meant creating new fields, changing their order on the page, setting default values, and so on. This meant the document page had to be entirely dynamically constructed in real time based on the customer's specifications. Before the project officially got started and before there was a database to work with, I created this page's UI, mostly to exercise the Manya platform I was creating. I mentored other developers who took over afterward.

One requirement was to store file attachments users uploaded with document records. Because our scale-up approach involves multiple web servers, it was impractical (and insecure) to store those files on publicly visible folders in the web app. I developed a reusable "file dump" service and component to move uploaded files into a dedicated database and easily pump them out to web clients on request.

One key feature of this new version of Quantifi would be its "list engine", which would show documented events selected using filters users could define themselves. The database designer for the project laid out the appropriate tables and started work on the list engine, while I implemented the list editor. When she fell ill for a few months, I took over construction of the list engine, which translates a given list definition into a set of stored procedures (technically, UDFs) in the database, mainly for performance reasons.

Although lists were an end in themselves, providing users with a natural work flow for day to day processing, they also served as input for Quantifi's new reporting engine. The database designer for the project and I collaborated on splitting up the work. She made a database component to generate the raw aggregate values and I made C# code to construct an in-memory data structure, calculate item percentages, subtotals, and totals, and construct a generic XML representation of the final data. In addition, I created the report viewer, which featured both a cross-tab data grid and line chart of the data. I created the charting component from scratch, too.

Sentri7's competitors had good products, many of which were more mature, but most suffered a significant limitation: it was cumbersome to create new filtered lists. Or customers had to pay their vendors to do it for them. Sentri7 made it fairly easy to define filtering rules and custom reports based on them.

The original plan for the S7 list editor was to have a highly dynamic, DHTML based "rules wizard" that would allow users to add one or more rules, selecting what to filter by and setting the specifics for each rule. Unfortunately, the first version was becoming an unwieldy mess of code and was not cross-browser compatible, so they rushed to production with a pure post-back approach. That is, each selection in a drop-down list or the like would cause the web page to be submitted and redrawn. I was brought into the project to revive the DHTML version. I rewrote it from scratch. The result was cleaner and much more compact and, most importantly, cross-browser compatible. All population of data based dynamically on selections was done using the AJAX model: XML RPCs back to a web service.

Pharmacy OneSource had already taken a step into their goal platfom -- ASP.NET / Sql Server -- by developing their newest product, Sentri7. But they had recognized that there were several key problems with it and that it was stretching the limits of the developers. One particular shortcoming at P1S was experience with DHTML / JavaScript / AJAX.

Initially, they hired me to fill the experience gap with client side web development, but realized that they needed something more. So they allowed me to develop Manya, a new UI framework, and Sara, a middle-tier framework. Together, Manya and Sara ("Manya" for short) represented a novel programming model that favored small amounts of linear, transparent code that decoupled UI from middle tier, but without a lot of the overhead that often comes with N-teir frameworks.

One key distinction is the Manya control set. Unlike standard ASP.NET controls, Manya's controls all had client-side components, which made for very easy JavaScript coding when that was required. Many also had built in remote procedure call (RPC) capabilities for fetching data, performing validation, and more via AJAX calls.

Manya also did away with the complex post-back model of traditional ASP.NET in favor of a model-view-controller (MVC) pattern, with all requests being channeled to explicit "operations" (e.g., "OpSave") or RPC handlers (e.g., "RpcCheckEmailAddress").

One other key distinction was the use of a "common information currency" that automatically coupled input from web forms and output to controls in a mutable collection that could be easily passed into and out of the middle tier and be used in all parts of the system. The mechanism made it possible to encode 90% of and application's validation rules in simple "domain objects" -- classes that have nothing but properties adorned with validation rule attributes like is-required or maximum value -- and let these data collections suck in these validation rules for use by the middle tier (e.g., during saves) and even the UI to do things like automatically decide maximum lengths for strings, for example.

One key goal for Manya was to implement the design patterns and styles poineered by the company's main business analyst / UI designer. Coders didn't need to put much thought into these requirements because they simply added controls to screens in the order indicated by designs and the style was inherently there. That is, the constraints Manya placed on the developer made the company's design patterns and standards an implicit, automatic thing instead of required a detailed, explicit effort on the developer's part.

Manya introduced a strong set of disciplines and tools for developing clear and concise code. One of its best strengths, in my opinion, was its openness to alternatives. My mantra in training was that one could bypass just about anything in Manya if it got in the way, but one rarely needed to.