<?xml version="1.0" ?>
      <rss
        xmlns:dc="http://purl.org/dc/elements/1.1/"
        xmlns:content="http://purl.org/rss/1.0/modules/content/"
        xmlns:atom="http://www.w3.org/2005/Atom"
        version="2.0"
      >
        <channel>
            <title><![CDATA[Kothar Labs]]></title>
            <link>https://kothar.net</link>
            <atom:link href="https://kothar.net/rss.xml" rel="self" type="application/rss+xml" />
            <description>
              <![CDATA[Projects and articles by Mike Houston]]>
            </description>
            <language>en</language>
            <lastBuildDate>Sat, 22 Aug 2026 00:00:00 GMT</lastBuildDate>
            
          <item>
            <title><![CDATA[Whalefish]]></title>
            <link>https://kothar.net/projects/whalefish</link>
            <pubDate>Sat, 22 Aug 2026 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/projects/whalefish</guid>
            <description>
              <![CDATA[<p>An experimental CMS built around a user-defined document taxonomy rather than fixed types like posts and pages.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>Whalefish was an experiment to build a flexible cloud-native CMS based on a user-defined document taxonomy. Each document type has a number of fields, which may be content, metadata or references to other documents in the taxonomy.</p>
<p>There’s no hard-coded concept of a <code>Post</code>, the document structure could represent anything from a blog, to a product catalog, to a photo gallery.</p>
<p>An initial version was started in 2016, built around similar technology to my blog and several other projects at the time: a Java web service implemented on Google App Engine, using the corresponding Google platform stack including Cloud Datastore. This attempt got fairly far, including document versioning, handling for multiple domains, preliminary hosting payment integration with Stripe, and an approval/publishing workflow. In the end it was the front-end elements which made the idea unworkable.</p>
<p>It had a set of mustache-based templates and layouts which could be composed in the administration UI to produce a rendered site from the taxonomy, but making this useable for end users was a challenge. Many low-code web design solutions such as Wix and Squarespace already exist, so I wasn’t going to do a better job with Whalefish.</p>
<p>When I decided to rebuild this blog I decided not to use what I’d been building for Whalefish and instead opted for a static Next.js generator with Markdown pages in Git. That’s been much more flexible for most of what I need personally from a content-based website, although there are still parts that are a bit clunky such as image handling.</p>
<p>I’ve since had another aborted attempt in 2020 to rebuild the idea on an AWS + Go stack, but that hasn’t got very far. I’d like to keep experimenting with this idea more, and it has evolved towards a headless API-first document store following the same taxonomy concept as the original, but dropping the complex templating system in favour of simplified pre-styled document rendering, leaving more complex layouts up to the implementor of the frontend.</p>
<p>This is not the only project where I have looked at concepts around document authoring and distribution.</p>
<ul>
<li><a href="communiscript">Communiscript</a> was an attempt at enabling co-authoring of structured technical documents by breaking the document down into individually versioned paragraphs with comments and revisions.</li>
<li><a href="kevalin/intro">Kevalin</a> is an ongoing experiment to enable peer-to-peer document publishing and referencing.</li>
</ul>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Communiscript]]></title>
            <link>https://kothar.net/projects/communiscript</link>
            <pubDate>Sat, 22 Aug 2026 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/projects/communiscript</guid>
            <description>
              <![CDATA[<p>Collaborative editing for structured documents, with each paragraph versioned separately and comments attached to the paragraph they refer to.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>Communiscript, later renamed Retype, was a collaborative structured document editing system. The idea was to treat a technical document like version-controlled source code: break it into sections and paragraphs, version each of those individually, and let several authors work on the same document at once with comments and revisions attached to the specific paragraph they refer to rather than to the document as a whole.</p>
<p>The original inspiration came from helping to write several academic papers, with each contributing author providing various sections and commenting on each other’s parts. I was also providing a final pass for English language usage (in a multinational research group the range of native languages was quite wide!). This was accomplished by emailing drafts back and forth, and reconciling edits in MS Word, an unpleasant task to say the least.</p>
<h2>Communiscript, pre-2012</h2>
<p>A 2018 backup of my old VPS turned up the original
Subversion repository, revealing how many attempts I made at this, each one starting again with a different technology stack, and each getting a bit further, but ultimately not making it to a useable product.</p>
<ul>
<li><strong>PHP and Zend Framework, September 2009 – January 2011.</strong> Plain PHP to begin
with, ported to Zend’s MVC over Christmas 2009. A jQuery front end using the
<em>pure</em> templating engine and jStore for offline storage, with Flash, Gears,
HTML5 and IE back ends — a good marker of the era. Login was OpenID. It ran as
a password-gated private alpha in late 2009. I have effectively no memory of this code.</li>
<li><strong>Java, EMF and GWT on App Engine, February – August 2011.</strong> A complete
rewrite. The domain model became an EMF <code>.ecore</code>, persisted first through
Teneo/Hibernate and then hand-annotated JPA on DataNucleus. Closure Templates
server-side, RestEasy for the API. It reached signup, profiles, document
creation and listing before it stopped.</li>
<li><strong>Renamed to Retype, August – September 2012.</strong> A third attempt, Maven and Jetty with
a GWT UiBinder editor and LESS, became <code>net.kothar.retype</code>.</li>
</ul>
<p>The model was the interesting part — <code>Document</code> → <code>DocumentRevision</code> → <code>Section</code>/<code>SectionRevision</code> → <code>Paragraph</code>/<code>ParagraphRevision</code>, where every revision was linked to its own parents, so the document’s history formed a graph. Comments were embedded against individual elements, authors and figures and tables were first-class domain entities, and editing happened against a <em>working revision</em> which could then be promoted to a published version. I was clearly never completely satisfied with the implementation of this idea though.</p>
<h2>Retype, March to October 2013</h2>
<p>Communiscript was renamed Retype, using the <code>rety.pe</code> domain, and the forth implementation was a slightly more serious attempt to make a working website.</p>
<p>I stuck with Java but ditched GWT for Velocity templates, and built the backend on MongoDB. Paragraph content was stored as Markdown and converted to HTML server-side for rendering, with editing handled in the browser by the Redactor WYSIWYG editor.</p>
<p>The frontend interactivity was written in Dart using the Polymer web components library. Neither has made it into mainstream use for web apps, with Dart mainly known for its association with Flutter, although web components are still an interesting technology I’d like to do more with.</p>
<h2>Retype, December 2015 to April 2016</h2>
<p>The fifth attempt kept the name and the concepts but was another rewrite, moving off Heroku and MongoDB onto Google App Engine. The data model was ported to the App Engine Datastore. Dart was abandoned, and the frontend went back to GWT, this time wrapping MediumEditor for the editing surface.</p>
<p>Two new ideas were added that the earlier versions didn’t have. Named branches, so a document revision could be worked on under a label rather than a single working copy, and cross-references (<code>xref</code>) on any element of a document, so a section or paragraph could be addressed by a stable human-readable name instead of a generated ID. Paragraphs also gained a content hash field, with the intention of making identical content shareable between revisions and the parent graph content-addressable in a similar way to Git, although I never got as far as populating it.</p>
<p>The awkward part was the translation between parallel models of the document on client and server. Since the datastore IDs didn’t translate cleanly into the client-side code, a lot of bookkeeping was needed to translate between the two.</p>
<p>This attempt fizzled out in 2016; I had at least some basic editing working, but the key collaboration features were still lacking.</p>
<h2>Current landscape (2026)</h2>
<p>There were several commercial tools offering a similar structured authoring experience when I was last working on this project, but most seem to have disappeared now (10 years later).</p>
<p>Most of the products in this area seem to be CRDT-style real-time collaborative editing like online MS Office applications, rather than fully structured write-review-amend authoring and editing tools. I suspect people make do with Office for simplicity, and most businesses won’t want to use a third-party tool to author documents that need to be shared as .docx files in the end anyway.</p>
<p>I did have an idea that sites publishing web novels or fan fiction might benefit from something like Communiscript, allowing inline comments and editing suggestions to crowd-source reviews as new chapters are published, but since I never got a fully working project up and running this hasn’t been explored.</p>
<h2>See also</h2>
<p>The same underlying interest has resurfaced in other projects since.</p>
<ul>
<li><a href="whalefish">Whalefish</a> approached documents from the publishing side, as a CMS built on a user-defined document taxonomy and schema.</li>
<li><a href="kevalin/intro">Kevalin</a> is an ongoing experiment in peer-to-peer document publishing and referencing.</li>
</ul>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[TQM]]></title>
            <link>https://kothar.net/projects/tqm</link>
            <pubDate>Sat, 18 Jul 2026 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/projects/tqm</guid>
            <description>
              <![CDATA[<p>A task queue service on DynamoDB and Lambda, with rate-limited dispatch, approximate priorities and task dependencies.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>Task Queue Manager (TQM) is an AWS-native task queue service built on DynamoDB and Lambda, deployable via Terraform. Influenced by SQS, Celery, BullMQ, and other task queuing systems, it’s designed for workloads where SQS falls short: specifically where you need fine-grained control over dispatch rate to downstream services, approximate priority ordering, or DAG-style task dependencies.</p>
<p>SQS’s Lambda integration handles polling for you, but this makes it hard to implement backpressure and circuit breaking without tuning concurrency and batch size - which is highly sensitive to Lambda execution time. TQM keeps you in control of dispatch, with rate limits defined by token buckets which can be used independently, or combined and shared by multiple queues.</p>
<h2>Design</h2>
<p>The data model builds on the DynamoDB priority queueing pattern described by the AWS Database Blog <a href="https://aws.amazon.com/blogs/database/implementing-priority-queueing-with-amazon-dynamodb/">“Implementing priority queueing with Amazon DynamoDB”</a>, using sparse GSIs and optimistic locking to implement visibility timeouts and approximate priority ordering without a dedicated queue service. Optional partition sharding supports higher-throughput queues. Tasks can be requeued at their original priority and timestamp rather than being penalised by going to the back of the queue - important for long-lived queues where jobs may wait days before dispatch.</p>
<p>Dependency handling follows a DAG model: tasks stay queued until their dependencies complete, with failure propagating up the tree or triggering a configurable failure handler.</p>
<p>Queue management is accessible either via a thin API layer or directly through the underlying tables, depending on how much abstraction you need.</p>
<p>This is an iteration on an <a href="/projects/stashmetrics/task-queue">earlier task queue implemented on top of MongoDB at Stashmetrics</a> around 2015. This supported the DAG-style dependency triggered workflow and priority-based queueing, but not the serverless architecture possible with DynamoDB and Lambda.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[ChilliOS]]></title>
            <link>https://kothar.net/projects/chillios</link>
            <pubDate>Sat, 18 Jul 2026 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/projects/chillios</guid>
            <description>
              <![CDATA[<p>A hobby x86 microkernel written in D around 2005, with content-addressed packages and object-based IPC.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>A hobby x86 32-bit OS kernel built collaboratively with a housemate around 2005, inspired by a university OS design course. Written in <a href="https://dlang.org/">D</a> - chosen for its object-oriented model and optional GC, which made it a better fit than C++ for bare-metal development. The bootloader used <a href="https://wiki.osdev.org/Rolling_Your_Own_Bootloader">ASM segments from the osdev.org wiki</a>; everything else was built from scratch, including a custom memory allocator and the D stdlib primitives needed in kernel mode (no libc available). The binary image was cross-compiled and loaded into QEMU.</p>
<h2>Design</h2>
<p>The kernel was designed as a microkernel built around non-blocking message passing - avoiding kernel interrupts for most IPC. Rather than a file-based abstraction, the design was object-based: OS primitives would expose interfaces that could be interacted with transparently over the network. This was influenced heavily by CORBA, which I had been working with during a placement year.</p>
<p>Data files were conceived as memory-mapped, content-addressed blocks. The content-addressing scheme was intended to enable verifiable package distribution pulled on demand from a DHT/CDN - informed by P2P file sharing software common at the time (pre-IPFS, early BitTorrent).</p>
<p>Something similar is implemented by the <a href="https://nixos.org/guides/how-nix-works/">Nix package manager</a>, although this is implemented in userspace and runs on several different host operating systems.</p>
<p><img src="/projects/Chillios.drawio.png" alt="ChilliOS basic architecture"></p>
<h2>Implementation status</h2>
<ul>
<li>Basic kernel memory allocator</li>
<li>Intel NIC driver - raw network packet send/receive</li>
<li>Framebuffer graphics driver with bitmap font rendering</li>
<li>Basic process management (no VMM or memory isolation)</li>
</ul>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Stashmetrics - Vector Search]]></title>
            <link>https://kothar.net/projects/stashmetrics/vector-search</link>
            <pubDate>Sat, 18 Jul 2026 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/projects/stashmetrics/vector-search</guid>
            <description>
              <![CDATA[<p>Finding profiles similar to a given input required k-nearest-neighbour search over sparse <a href="tfidf">TF-IDF term vectors</a>. The initial approach used a Lucene search index with cosine distance ranking - this produced good results but took on the order of 10 seconds per query, which was too slow for interactive use.</p>
<h2>BK-tree index</h2>
<p>The solution was a custom BK-tree implementation written in Go. A <a href="https://en.wikipedia.org/wiki/BK-tree">BK-tree</a> is a metric tree structure that supports efficient nearest-neighbour queries under any distance metric satisfying the triangle inequality. Here, distance was computed over sets of TF-IDF terms (sparse term sets), enabling fast approximate matching without needing to compare every candidate.</p>
<h3>Index structure</h3>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>Finding profiles similar to a given input required k-nearest-neighbour search over sparse <a href="tfidf">TF-IDF term vectors</a>. The initial approach used a Lucene search index with cosine distance ranking - this produced good results but took on the order of 10 seconds per query, which was too slow for interactive use.</p>
<h2>BK-tree index</h2>
<p>The solution was a custom BK-tree implementation written in Go. A <a href="https://en.wikipedia.org/wiki/BK-tree">BK-tree</a> is a metric tree structure that supports efficient nearest-neighbour queries under any distance metric satisfying the triangle inequality. Here, distance was computed over sets of TF-IDF terms (sparse term sets), enabling fast approximate matching without needing to compare every candidate.</p>
<h3>Index structure</h3>
<p>Indexing ran offline as a nightly batch job. Each indexed record (user profile or content item) was represented by its top-10 TF-IDF terms. To keep tree sizes manageable, a separate BK-tree was built per keyword: for each keyword, all records with that keyword in their top-10 terms were selected and a tree was built over their full sparse term vectors. Trees were serialised to binary gob format and stored on S3.</p>
<h3>Query execution</h3>
<p>At query time:</p>
<ol>
<li>The three highest-weighted keywords from the input vector (or manually entered) were selected.</li>
<li>The corresponding BK-trees were fetched from S3 and deserialised.</li>
<li>A k-nearest-neighbour search was run independently on each tree.</li>
<li>Results from all three trees were merged and re-ranked against the original input vector.</li>
</ol>
<p>This approach reduced query latency from ~10 seconds (Lucene + cosine distance) to tens-to-hundreds of milliseconds - fast enough for interactive search with the infrastructure budget available.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Stashmetrics - TF-IDF Analysis]]></title>
            <link>https://kothar.net/projects/stashmetrics/tfidf</link>
            <pubDate>Sat, 18 Jul 2026 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/projects/stashmetrics/tfidf</guid>
            <description>
              <![CDATA[<p>TF-IDF (Term Frequency–Inverse Document Frequency) was the foundation of Stashmetrics’ content analysis pipeline, applied to three sources: user profile bios, post content, and the text of external links shared in posts.</p>
<h2>Baseline and term extraction</h2>
<p>Rather than computing IDF against the indexed corpus alone, we used a sample of the Twitter firehose as a baseline - scoring terms against what was common across the platform as a whole. This meant that terms salient to a specific niche community would score highly even if they appeared frequently within that community, as long as they were relatively uncommon in general discourse.</p>
<p>The top-N terms extracted per user or post formed a sparse term vector representing that profile or piece of content.</p>
<h2>Community grouping and topic modelling</h2>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>TF-IDF (Term Frequency–Inverse Document Frequency) was the foundation of Stashmetrics’ content analysis pipeline, applied to three sources: user profile bios, post content, and the text of external links shared in posts.</p>
<h2>Baseline and term extraction</h2>
<p>Rather than computing IDF against the indexed corpus alone, we used a sample of the Twitter firehose as a baseline - scoring terms against what was common across the platform as a whole. This meant that terms salient to a specific niche community would score highly even if they appeared frequently within that community, as long as they were relatively uncommon in general discourse.</p>
<p>The top-N terms extracted per user or post formed a sparse term vector representing that profile or piece of content.</p>
<h2>Community grouping and topic modelling</h2>
<p>Community discovery was graph-based: follower relationships and shared list membership were traversed starting from a keyword-seeded set of profiles, expanding outward through the social graph to identify a connected community. Once a community was identified, TF-IDF term vectors for its members were aggregated and k-means clustering applied to identify the topics and themes most characteristic of that group. This allowed the platform to surface content that was popular specifically within a niche rather than relying on platform-wide popularity signals.</p>
<h2>Building vectors</h2>
<p>The extracted TF-IDF terms served double duty: they were both the basis for community clustering and the input to the <a href="vector-search">vector similarity search</a> pipeline. Each profile and content item was represented as a sparse set of its most salient terms, which could then be indexed and queried efficiently using a BK-tree structure.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Stashmetrics - Task Queue]]></title>
            <link>https://kothar.net/projects/stashmetrics/task-queue</link>
            <pubDate>Sat, 18 Jul 2026 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/projects/stashmetrics/task-queue</guid>
            <description>
              <![CDATA[<p>The Stashmetrics data pipeline was driven by a custom task queue built on MongoDB, supporting DAG-style task dependencies and priority-based dispatch.</p>
<h2>Design</h2>
<p>The core processing engine was <a href="https://storm.apache.org/">Apache Storm</a>, which handled event streaming across a topology of bolts. This allowed fan-out (splitting work across parallel sub-tasks) and fan-in (aggregating results before triggering downstream steps), making it natural to model multi-stage workflows - for example, fetching multiple user profiles in parallel and triggering a report update once they had all completed.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>The Stashmetrics data pipeline was driven by a custom task queue built on MongoDB, supporting DAG-style task dependencies and priority-based dispatch.</p>
<h2>Design</h2>
<p>The core processing engine was <a href="https://storm.apache.org/">Apache Storm</a>, which handled event streaming across a topology of bolts. This allowed fan-out (splitting work across parallel sub-tasks) and fan-in (aggregating results before triggering downstream steps), making it natural to model multi-stage workflows - for example, fetching multiple user profiles in parallel and triggering a report update once they had all completed.</p>
<p>The MongoDB task queue sat alongside the Storm topology to handle two cases that didn’t fit cleanly into a streaming graph: rate-limited tasks that needed to be held back and dispatched at a controlled pace, and high-priority tasks (such as real-time user-triggered searches) that needed to jump ahead of pending background indexing work. Tasks could declare dependencies on other tasks and remained queued until all dependencies had completed, allowing the pipeline to chain async operations and API calls across Storm and queue boundaries.</p>
<h2>More recent work</h2>
<p>I’m still tinkering with some of these ideas in <a href="/projects/tqm">TQM</a>, a serverless reimplementation using DynamoDB and Lambda. TQM preserves the DAG dependency model and priority ordering while adding rate limiting, backpressure, and partition sharding - capabilities that were not needed at Stashmetrics’ scale but are the primary motivation for the newer design.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Getting my grocery receipts into ChatGPT the long way round]]></title>
            <link>https://kothar.net/blog/2026/mcp-grocery-receipts</link>
            <pubDate>Sun, 22 Feb 2026 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/2026/mcp-grocery-receipts</guid>
            <description>
              <![CDATA[<p>My grocery purchase history from online orders over the last few years is sitting in a database, ready for analysis with an LLM - but getting it there turned into a surprisingly interesting journey through the Model Context Protocol (MCP).</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>My grocery purchase history from online orders over the last few years is sitting in a database, ready for analysis with an LLM - but getting it there turned into a surprisingly interesting journey through the Model Context Protocol (MCP).</p>
<p>I’ve been working on a project called <em>Crosscheckout</em> to collect and analyze online grocery orders. It ingests forwarded receipt emails, parses the attached PDFs, and stores line-item data in a database. It’s a serverless application implemented in TypeScript on AWS Lambda, using SES for ingestion and DynamoDB for storage.</p>
<pre><code>specification {
  element person {
    style {
      color amber
    }
  }
  element system {
    style {
      color green
    }
  }
  element component
  element database {
    style {
      shape storage
      color green
    }
  }
  element module {
    style {
      opacity 15%
    }
  }
}
model {
  shopper = person 'Shopper'
  retailer = system 'Retailer'
  crosscheckout = module 'Crosscheckout' {
    ses = system 'SES'
    db = database 'Database'
    webapp = component 'Web App'
    emailProcessor = component 'Email Processor'
    pdfParser = component 'PDF Parser'
    api = component 'API'

    emailProcessor -> pdfParser
    pdfParser -> db
    api -> pdfParser
    api -> db
    ses -> emailProcessor
    webapp -> api
  }

  shopper -> crosscheckout.webapp
  retailer -> ses
}
views {
  view index  {
    title "Crosscheckout"
    description """
      Overview of Crosscheckout architecture
    """
    include *, crosscheckout.*
    autoLayout LeftRight
  }
}
</code></pre>
<p>The app already gives me several useful views like:</p>
<ul>
<li>frequently bought items,</li>
<li>product price history, and</li>
<li>the cost of an “average shop”.</li>
</ul>
<p>That gave me a nice overview of my shopping history, but I still wanted a more conversational layer for questions that are awkward to pre-build into dashboards.</p>
<ul>
<li>“What meals could I make from what I bought this week? I have used the chicken”</li>
<li>“How often should I subscribe to coffee so I don’t run out?”</li>
<li>“What do I usually spend on fresh vs packaged food?”</li>
</ul>
<p>Since I already had the data stored and organised, it felt like I should be able to get an LLM chatbot to query it and answer these questions for me. MCP is an open protocol introduced by Anthropic in 2024 which gives agents access to new tools that they can invoke over HTTP to fetch and interact with data from other systems, so I thought it would be a fun experiment to see if I could add an MCP layer for Crosscheckout.</p>
<h2>Building an MCP endpoint on Lambda</h2>
<p>Crosscheckout is written in TypeScript and runs on AWS Lambda, so I started from the TypeScript MCP SDK.
This <a href="https://github.com/awslabs/run-model-context-protocol-servers-with-aws-lambda/tree/main" title="Run Model Context Protocol (MCP) servers with AWS Lambda">awslabs guide for MCP on Lambda</a> was also helpful, illustrating how to implement a lambda MCP handler, but I didn’t want to wrap an existing ‘local’ service. I wanted the Lambda itself to implement MCP.</p>
<p>Getting that part working was fairly straightforward, and some prototyping with Claude Opus got something working in an hour or so.</p>
<p>The lambda itself hosts an express server containing the MCP SDK server, and then implements a shim for handling an API Gateway proxy integration request and forwarding it to express.</p>
<pre><code>specification {
  element agent {
    style {
      icon bootstrap:robot
      color amber
    }
  }
  element system {
    style {
      color green
    }
  }
  element component
  element database {
    style {
      shape storage
      color green
    }
  }
  element module {
    style {
      opacity 15%
    }
  }
}
model {
  agent = agent 'AI Agent'
  apigw = system 'API Gateway'
  db = database 'Database'
  mcplambda = module 'MCP Lambda' {
    handler = component 'Lambda Handler'
    express = component 'Express Server'
    mcp = component 'MCP SDK'
    tools = component 'Tool implementations' {
      style {
        multiple true
      }
    }

    handler -> express
    express -> mcp
    mcp -> tools
  }
  oauthlambda = component 'OAuth Lambda'
  cognito = system 'Cognito'

  agent -> apigw
  apigw -> mcplambda.handler
  mcplambda.tools -> db
  apigw -> oauthlambda
  oauthlambda -> cognito
}
views {
  view index  {
    title "Lambda MCP"
    description """
      Overview of components needed to implement an MCP
      endpoint on AWS Lambda
    """
    include *, mcplambda.*
    autoLayout LeftRight
  }
}
</code></pre>
<p>I exposed five tools, matching the existing data queries implemented for the web app’s report views. It’s extremely likely that an alternative query interface would be more flexible and allow more targeted queries from agents, but these tools reflect the current conceptual model of the data.</p>
<ul>
<li><code>list_receipts</code></li>
<li><code>get_receipt</code></li>
<li><code>get_price_history</code></li>
<li><code>get_frequently_bought</code></li>
<li><code>get_analysis</code></li>
</ul>
<h2>Debugging transport and platform behaviour</h2>
<h3>OAuth</h3>
<p>Connecting Cursor as a first client was straightforward. Once it could reach the endpoint, tool calls worked and I could query my shopping data immediately.</p>
<p>ChatGPT was not so simple.</p>
<p>I initially had enough auth to make Cursor happy, but not enough for ChatGPT. I had to add OAuth discovery endpoints so it could register as a client and complete the full flow.
After that, ChatGPT reported “Connected to Crosscheckout” but nothing appeared in the UI, and I saw no real tool activity.</p>
<h3>MCP Transports</h3>
<p>The next rabbit hole was transport support.
ChatGPT itself kept nudging me toward SSE or websockets as suitable transports. Various forum posts I found also suggested that it only supported SSE, but websockets was definitely not an option as it’s still only at <a href="https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1288" title="SEP-1288: WebSocket Transport (MCP proposal)">the proposal stage</a>.</p>
<p>SSE (server-sent events) are not at first glance a good fit for Lambda, which historically has been a request/response model, however since November 2025 API Gateway has added <a href="https://aws.amazon.com/blogs/compute/building-responsive-apis-with-amazon-api-gateway-response-streaming/" title="Building responsive APIs with Amazon API Gateway response streaming">support for response streaming</a> and explicitly calls out SSE as a use case. Unfortunately ChatGPT didn’t know about this, and insisted I would need to use the slightly earlier support for <a href="https://aws.amazon.com/blogs/compute/introducing-aws-lambda-response-streaming/" title="Introducing AWS Lambda response streaming">streaming via Lambda Function URLs</a> or switch to a container-based model.</p>
<p>Ignoring that advice, I pointed Opus at some examples and it added the necessary changes to support streamed responses and implemented the SSE MCP protocol as a fallback.</p>
<h3>Empty streams</h3>
<p>Then all streamed responses were empty - the request was successful, but always returned a zero-byte body. Surely a bug in the newly added streaming handler?</p>
<p>As is often the case, the cause was not immediately obvious and turned out to be unrelated to the implementation of streaming in the Lambda handler. After much manual debugging I located the culprit: my API Gateway deployment trigger conditions didn’t include the streaming config change, so I was testing stale infra.</p>
<p>Once fixed, both transports were working. At this point I also noticed an operational quirk of Lambda for this use case: since each request closes the connection almost immediately, the MCP client tries to reconnect after 1 second to receive any new push events from the server. If I actually wanted to keep the connection open and send push events this wouldn’t be ideal due to Lambda’s instance-per-request model, but in this case the solution I’ve settled on is to pass a <code>retry: 30000</code> (retry after 30s) back to the client just before closing the connection. This postpones the next polling call and prevents a huge number of pointless calls being made to the endpoint.</p>
<h2>Plus plan required</h2>
<p>After all the protocol and infrastructure work, the thing blocking me from connecting to ChatGPT was account tier.
Custom MCP servers in ChatGPT require at least a Plus/Pro plan. The free tier won’t activate the integration in practice, despite offering the option in the settings panel with no indication that it wouldn’t work.</p>
<p>I upgraded my plan but still had flaky behavior for a while. <a href="https://platform.openai.com/chat/edit">OpenAI’s Prompt Builder</a> worked straight away, and I eventually added a <code>resources/list</code> handler plus <code>application/octet-stream</code> support because I could see from logs that ChatGPT was requesting both but receiving error responses.</p>
<p>Roughly an hour later (and a lot of refreshing and re-testing), the app appeared and stayed connected. I was finally able to ask about my apricot stock levels!</p>
<p><img src="/static/2026/crosscheckout_apricots.png" alt="When did I last buy apricots?"></p>
<h2>Was it worth it?</h2>
<p>Once the MCP service finally worked, I could ask useful follow-up questions about my own shopping history.
If my goal had been only “get grocery data into an LLM once,” then this probably wouldn’t have been worth the effort. I could have just dumped the PDFs into a ChatGPT or NotebookLM project and asked questions about them with no extra work.</p>
<p>But as an engineering exercise it was worth it. It’s been fun to try out some new concepts around MCP, and to play with API Gateway streaming behavior. It’s also been instructive in prompting LLMs to generate some of these implementations when the latest version of the spec was clearly not available before the model training data was captured.</p>
<h2>Sampling Bias</h2>
<p>The most interesting part has been using the model to identify data gaps and suggest better analyses. For example, asking which products I should buy to allow me to make a wider variety of meals led to ChatGPT identifying that there were no staples like pasta or rice in my ‘frequently bought items’ and recommending I buy some spaghetti. This is sampling frame bias however: the data represents purchase history, but not the current state of my supplies, since I don’t buy longer-life products as often as fresh vegetables with short shelf lives.</p>
<p>Once pointed out, the LLM was able to do a better job of identifying more items in my shopping history, and one suggestion I want to explore at some point is “cuisine entropy” as a way to improve recommendation variety. This <a href="https://medium.com/@reika.k.fujimura/entropy-sampling-how-to-balance-relevance-and-surprise-in-recommendation-2223417a38ce" title="Entropy Sampling — How to balance “relevance” and “surprise” in recommendation">article on entropy sampling by Reika Fujimura</a> gives a good introduction to similar ideas in recommendation algorithms.</p>
<hr>
<h2>Code</h2>
<p>I’ve not released the code for Crosscheckout at this time, but I do intend to eventually. If you’re interested feel free to give me a shout at labs@kothar.net</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Managing Multiple Git Identities with Conditional Includes]]></title>
            <link>https://kothar.net/blog/2025/directory-targeted-git-config</link>
            <pubDate>Fri, 09 May 2025 09:52:00 GMT</pubDate>
            <guid>https://kothar.net/blog/2025/directory-targeted-git-config</guid>
            <description>
              <![CDATA[<p>If you contribute to both internal and client Git repositories - or juggle multiple client projects - keeping your
commit authors and SSH credentials in order can get messy fast. Thankfully, Git offers a powerful feature to help with
that: <strong>conditional includes</strong>.</p>
<p>In this post, I’ll walk through how I use this approach to keep my Git identity and credentials scoped correctly
depending on the project folder.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>If you contribute to both internal and client Git repositories - or juggle multiple client projects - keeping your
commit authors and SSH credentials in order can get messy fast. Thankfully, Git offers a powerful feature to help with
that: <strong>conditional includes</strong>.</p>
<p>In this post, I’ll walk through how I use this approach to keep my Git identity and credentials scoped correctly
depending on the project folder.</p>
<h2>Why Use Conditional Includes?</h2>
<p>By default, Git uses global config settings from <code>~/.gitconfig</code>, which applies to <em>every</em> repo. But when working across
multiple organisations or contexts, you often need different:</p>
<ul>
<li>Commit author name and email</li>
<li>GPG/SSH signing keys</li>
<li>SSH command settings (e.g. using a different key for GitHub)</li>
</ul>
<p>Using <code>includeIf</code> blocks in <code>.gitconfig</code>, you can automatically load extra config <em>only</em> when working in specific
folders.</p>
<h2>The Setup</h2>
<h3>Global <code>.gitconfig</code></h3>
<p>Here’s what my base <code>~/.gitconfig</code> looks like:</p>
<pre><code>[user]
	name = Mike Houston
	email = mike@kothar.net

[init]
	defaultBranch = main

[pull]
	rebase = true

[core]
	autocrlf = input
	excludesfile = /Users/Michael.Houston/.gitignore_global

[includeIf "gitdir:~/acme-team/"]
	path = .gitconfig-acme-team

[includeIf "gitdir:~/internal/"]
	path = .gitconfig-internal
</code></pre>
<p>The key part is this block:</p>
<pre><code>[includeIf "gitdir:~/acme-team/"]
	path = .gitconfig-acme-team
</code></pre>
<p>This tells Git:</p>
<blockquote>
<p>“If I’m working in a repo under <code>~/acme-team/</code>, also load settings from <code>~/.gitconfig-acme-team</code>.”</p>
</blockquote>
<p>Repeat that pattern for other directories where you need distinct settings.</p>
<h2>Team-Specific Config</h2>
<p>Here’s what I’ve got in <code>~/.gitconfig-acme-team</code>:</p>
<pre><code>[user]
	name = Mike Houston
	email = mike@acme.org
	signingkey = ~/.ssh/acme-github-signing-key

[core]
	sshCommand = ssh -i ~/.ssh/id_ecdsa_acme

[commit]
	gpgsign = true

[gpg]
	format = ssh
</code></pre>
<p>This sets:</p>
<ul>
<li>My name and Acme team email</li>
<li>The GPG signing key to use (backed by SSH)</li>
<li>A custom <code>sshCommand</code> so the right key is used when authenticating with remote repos</li>
<li>Ensures commits in these repos are signed</li>
</ul>
<h2>Best Practices</h2>
<ul>
<li><strong>Paths matter</strong>: The <code>gitdir:</code> value must end with a slash and point to a parent directory, not a specific repo. e.g.
<code>~/acme-team/</code>, not <code>~/acme-team/my-repo</code></li>
<li><strong>Use <code>~</code> safely</strong>: Git expands <code>~</code> correctly in paths</li>
<li><strong>Keep configs modular</strong>: One include file per context is cleanest</li>
<li><strong>Version control them</strong> (carefully): You might want to keep these in a dotfiles repo - just avoid committing anything
sensitive like private key paths</li>
</ul>
<h2>Final Thoughts</h2>
<p>This pattern has saved me tons of context switching headaches and reduced the risk of pushing commits with the wrong
author info or SSH key. It’s flexible, clean, and just works.</p>
<p>If you frequently shift between clients or internal repos, setting this up is well worth the few minutes of config
fiddling.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[CSView 1.3.8 (Windows x64)]]></title>
            <link>https://kothar.net/blog/csview-1.3.8</link>
            <pubDate>Mon, 03 Feb 2025 22:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/csview-1.3.8</guid>
            <description>
              <![CDATA[<p>A new build of CSView is available for Windows. This is a maintenance release updating the project dependencies to the latest versions, and fixing a minor bug relating to scroll bar handling.</p>
<p>The new version is 1.3.8 and is <a href="/csview/">available to download</a></p>
<p>Version 2.0 for macOS is getting close to a release. Apologies to anyone who’s been waiting for this for a long time! It will be a paid application, although my current plan is that it will be a minimal cost one-time purchase from the App Store. Version 1.x will continue to be available as an open-source project for all currently supported platforms.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>A new build of CSView is available for Windows. This is a maintenance release updating the project dependencies to the latest versions, and fixing a minor bug relating to scroll bar handling.</p>
<p>The new version is 1.3.8 and is <a href="/csview/">available to download</a></p>
<p>Version 2.0 for macOS is getting close to a release. Apologies to anyone who’s been waiting for this for a long time! It will be a paid application, although my current plan is that it will be a minimal cost one-time purchase from the App Store. Version 1.x will continue to be available as an open-source project for all currently supported platforms.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[CSView 2.0 Beta]]></title>
            <link>https://kothar.net/blog/csview_2_0_beta_1</link>
            <pubDate>Sat, 07 Oct 2023 11:55:00 GMT</pubDate>
            <guid>https://kothar.net/blog/csview_2_0_beta_1</guid>
            <description>
              <![CDATA[<p>CSView 2.0 Beta for MacOS is now
<a href="https://testflight.apple.com/join/iJBXwzSW">available on TestFlight</a>.</p>
<p>Version 2.0 still doesn’t support all the features provided in CSView
1.x, however it’s at a point where I find it usable and want to get it
into other people’s hands to see where any critical improvements are
needed.</p>
<p>See the <a href="/blog/csview_2_0_alpha_1">Alpha release announcement</a> for
full details of the changes in CSView 2.0</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>CSView 2.0 Beta for MacOS is now
<a href="https://testflight.apple.com/join/iJBXwzSW">available on TestFlight</a>.</p>
<p>Version 2.0 still doesn’t support all the features provided in CSView
1.x, however it’s at a point where I find it usable and want to get it
into other people’s hands to see where any critical improvements are
needed.</p>
<p>See the <a href="/blog/csview_2_0_alpha_1">Alpha release announcement</a> for
full details of the changes in CSView 2.0</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[CSView 1.3.6]]></title>
            <link>https://kothar.net/blog/csview-1.3.6</link>
            <pubDate>Sun, 01 May 2022 22:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/csview-1.3.6</guid>
            <description>
              <![CDATA[<p>A new build of CSView is available, and it now supports Apple Silicon Macs (i.e. M1 series of SoCs which use the ARM aarch64 architecture)</p>
<p>The new version is 1.3.6 and is <a href="/csview/">available to download</a></p>
<p>Due to Java’s support for Apple Silicon, it’s not possible to build an app bundle which supports both Apple and Intel CPUs at present. This is a requirement for distributing the app through the App Store, so for now I am only able to distribute the Intel build of CSView there.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>A new build of CSView is available, and it now supports Apple Silicon Macs (i.e. M1 series of SoCs which use the ARM aarch64 architecture)</p>
<p>The new version is 1.3.6 and is <a href="/csview/">available to download</a></p>
<p>Due to Java’s support for Apple Silicon, it’s not possible to build an app bundle which supports both Apple and Intel CPUs at present. This is a requirement for distributing the app through the App Store, so for now I am only able to distribute the Intel build of CSView there.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[CSView 2.0 Alpha 1]]></title>
            <link>https://kothar.net/blog/csview_2_0_alpha_1</link>
            <pubDate>Mon, 25 May 2020 11:55:00 GMT</pubDate>
            <guid>https://kothar.net/blog/csview_2_0_alpha_1</guid>
            <description>
              <![CDATA[<p>CSView 2.0 Alpha 1 for MacOS is now available for download!</p>
<p><a href="/assets/CSView_2.0_Alpha_1.zip">Download CSView 2.0 Alpha 1</a> - macOS
10.12+, 64-bit Intel, 3.5MB</p>
<p>Version 2.0 is a macOS native application, bringing performance and
visual improvements.</p>
<ul>
<li>Startup time is improved over the 1.x implementation</li>
<li>UI rendering is now able to take advantage of native graphics
acceleration</li>
<li>Dark/Light theme is picked up from your OS settings</li>
<li>macOS UI features such as tabbed document windows come for free</li>
<li>Larger number of rows supported (rendering tested with 10B+ rows and
columns)</li>
<li>More opportunity for future performance improvements and new
features</li>
</ul>
<p>Alpha 1 doesn’t yet support all the features of the 1.x releases, such
as search and custom column delimiters. These will be coming in future
alphas before the official 2.0 release.</p>
<p>I have a prototype grid renderer in the works for Windows too, but I’m
focusing on getting a fully working Mac release ready first.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>CSView 2.0 Alpha 1 for MacOS is now available for download!</p>
<p><a href="/assets/CSView_2.0_Alpha_1.zip">Download CSView 2.0 Alpha 1</a> - macOS
10.12+, 64-bit Intel, 3.5MB</p>
<p>Version 2.0 is a macOS native application, bringing performance and
visual improvements.</p>
<ul>
<li>Startup time is improved over the 1.x implementation</li>
<li>UI rendering is now able to take advantage of native graphics
acceleration</li>
<li>Dark/Light theme is picked up from your OS settings</li>
<li>macOS UI features such as tabbed document windows come for free</li>
<li>Larger number of rows supported (rendering tested with 10B+ rows and
columns)</li>
<li>More opportunity for future performance improvements and new
features</li>
</ul>
<p>Alpha 1 doesn’t yet support all the features of the 1.x releases, such
as search and custom column delimiters. These will be coming in future
alphas before the official 2.0 release.</p>
<p>I have a prototype grid renderer in the works for Windows too, but I’m
focusing on getting a fully working Mac release ready first.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[MacOS Catalina - Notarization and App Store Signing with Java 11]]></title>
            <link>https://kothar.net/blog/macos_catalina_java_11</link>
            <pubDate>Sun, 13 Oct 2019 16:55:00 GMT</pubDate>
            <guid>https://kothar.net/blog/macos_catalina_java_11</guid>
            <description>
              <![CDATA[<p>With the release of macOS Catalina (10.15) there have been some changes to the way software needs to be signed. In particular, Apple has introduced a new <a href="https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution">Notarization</a> service which allows developers to pre-register the digital signature of an application so that it can be later checked by the operating system when a user downloads software outside the App Store.</p>
<p>I took this as a good opportunity to update to Java 11, and the rest of this article covers the changes I’ve needed to make to the packaging process.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>With the release of macOS Catalina (10.15) there have been some changes to the way software needs to be signed. In particular, Apple has introduced a new <a href="https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution">Notarization</a> service which allows developers to pre-register the digital signature of an application so that it can be later checked by the operating system when a user downloads software outside the App Store.</p>
<p>I took this as a good opportunity to update to Java 11, and the rest of this article covers the changes I’ve needed to make to the packaging process.</p>
<p>Earlier versions of macOS still required software to be signed by the developer to allow it to run on a user’s system, however notarization takes this one step further, and presumably allows Apple to disable malicious applications when macOS phones home to check if the app has been notarized.</p>
<p>If you’re using Xcode for development and packaging of your app, this is all handled for you in the latest versions. Unfortunately, CSView is written in Java, requiring a custom build and packaging process, and this process has not survived the update to Catalina. It appears that some of the existing configuration was not compatible with the new sandboxing requirements in Catalina, so simply notarizing the old versions of the app was not sufficient, even for versions distributed through the App Store.</p>
<h2>Packaging a Java 11 macOS App</h2>
<p>Java 8 included a tool called <a href="https://docs.oracle.com/javase/9/tools/javapackager.htm#JSWOR719">javapackager</a> which allowed an application written in Java to be bundled into an installable package on several different operating systems, including macOS. This was the approach I originally took with CSView and it worked well, creating both a .dmg file containing the bundled application and a signed package ready for submission to the App Store. Unfortunately this tool was removed in version 11 of the JDK, with the replacement tool <a href="https://openjdk.java.net/jeps/343">jpackage</a> still in development and not provided as part of the standard distributions of the JDK (it may be included with Java 14).</p>
<p>Following the guide <a href="https://medium.com/@adam_carroll/java-packager-with-jdk11-31b3d620f4a8">Using the Java Packager with JDK 11</a> by Adam Carroll, I was able to get the packaging working again with Java 11. This has the nice benefit that with the post Java 9 support for modules, the app bundle is now about 30% smaller.</p>
<h2>Signing and Notarizing for macOS Catalina</h2>
<p>Unfortunately the available versions of the tool have not yet been updated to deal with the new more stringent signing requirements for Catalina. This means that I have needed to replace the signing step supported by <code>jpackage</code>, and to then generate the app installation packages manually from that signed app.</p>
<p>Several hours of experimentation later, and I have arrived at the following changes needed for Catalina:</p>
<ol>
<li>Enable the hardened runtime when signing. This is done by adding the <code>-options runtime</code> flag to <code>codesign</code></li>
<li>Enable the secure timestamp when signing, using the <code>-timestamp</code> flag</li>
<li>Add some exceptions to the sandbox using the entitlements file. The ones which seem to be needed are:</li>
</ol>
<pre><code>   &#x3C;key>com.apple.security.cs.allow-jit&#x3C;/key>
   &#x3C;true/>
   &#x3C;key>com.apple.security.cs.allow-unsigned-executable-memory&#x3C;/key>
   &#x3C;true/>
   &#x3C;key>com.apple.security.cs.disable-executable-page-protection&#x3C;/key>
   &#x3C;true/>
   &#x3C;key>com.apple.security.cs.allow-dyld-environment-variables&#x3C;/key>
   &#x3C;true/>
   &#x3C;key>com.apple.security.cs.disable-library-validation&#x3C;/key>
   &#x3C;true/>
</code></pre>
<p>The signing command now looks like this:</p>
<pre><code>codesign --force --deep \
	--options runtime \
	--timestamp \
	--prefix [SIGNING PREFIX] \
	--entitlements [APP ENTITLEMENTS FILE] \
	--sign "[DEVELOPER SIGNING KEY]" \
	"[FILE TO SIGN]"
</code></pre>
<p>You’ll need sign all dylib files and executables in your app.</p>
<p>Once your app is signed, you can then use the <code>productbuild</code> command to build a package for the app store. The command should look something like this:</p>
<pre><code>productbuild --component [APP BUNDLE] /Applications \
	--sign "[INSTALLER SIGNING KEY]" \
	--product [APP BUNDLE]/Contents/Info.plist \
	[OUTPUT PACKAGE]
</code></pre>
<h2>Which keys should I use?</h2>
<ul>
<li>If you’re building a .dmg or .pkg file for distribution outside the app store, you’ll need to use the <code>Developer ID Application</code> and <code>Developer ID Installer</code> keys. These will be accepted by the Notarization service.</li>
<li>If you’re building a .pkg file for upload to the app store, you should use the <code>3rd Party Mac Developer Application</code> and <code>Installer</code> keys. These will be accepted by App Store Connect when uploading your app.</li>
</ul>
<p>Note that you don’t need to notarize your app if you’re distributing it via the App Store, it’s just for apps being distributed directly to users. In fact, the notarization service will complain that the ‘3rd Party’ keys are not valid for signing the app.</p>
<h2>Notarization</h2>
<p>Notarization requires that you upload your signed app package to Apple, who will check the app.  You can perform the upload from the command line using the <code>xcrun altool</code> command:</p>
<pre><code>xcrun altool --notarize-app \
	--primary-bundle-id [BUNDLE ID] \
	--file [DMG FILE] \
	-u [APP STORE CONNECT USERNAME] \
	-p [APP STORE CONNECT PASSWORD]
</code></pre>
<p>Once notarization is complete you can check the results using the same tool:</p>
<pre><code>xcrun altool --notarization-info \
	[NOTARIZATION UUID] \
	-u [APP STORE CONNECT USERNAME] \
	-p [APP STORE CONNECT PASSWORD]
</code></pre>
<h2>App Store Connect Upload</h2>
<p>There used to be a tool called Application Loader which was used to upload a package to App Store Connect. This has now been removed in Xcode 11, and instead the same <code>altool</code> command line tool is used as with notarization:</p>
<pre><code>xcrun altool -upload-app -f [APP STORE PACKAGE] \
	-u [APP STORE CONNECT USERNAME] \
	-p [APP STORE CONNECT PASSWORD]
</code></pre>
<p>The password in the above commands will need to be a single-use application password if you’re using MFA on your App Store Connect account, which I think is now compulsory.  You can store the password in your keychain to avoid including it in build scripts. For more information, take a look at the <a href="https://developer.apple.com/documentation/xcode/notarizing_your_app_before_distribution/customizing_the_notarization_workflow#3087734">Customizing the Notarization Workflow</a> article on the Apple developer website.</p>
<h2>Conclusion</h2>
<p>That’s it! Hopefully these notes will help someone else avoid searching around the details scattered across several sites and articles. Let me know if this does/doesn’t work for you, or if you have any improvements to the process.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Streaming multipart HTTP requests in Go]]></title>
            <link>https://kothar.net/blog/streaming_multipart_http</link>
            <pubDate>Sat, 23 Feb 2019 16:13:00 GMT</pubDate>
            <guid>https://kothar.net/blog/streaming_multipart_http</guid>
            <description>
              <![CDATA[<p>Having seen Peter Bourgon’s post
on <a href="https://peter.bourgon.org/blog/2019/02/12/multipart-http-responses.html">multipart http responses</a> come up in the
Golang Weekly newsletter, I thought I would write some notes on a related problem I was looking at recently: how to
stream a multipart upload to a remote HTTP server.</p>
<p>My specific use case is to receive a stream from AWS S3, and pipe it into the upload API for Asana. It’s probably ending
up back in S3, but that’s neither here nor there.</p>
<p>The multipart writer provided with Go’s standard library makes serving multipart responses really easy, but sending an
HTTP request needs an io.Reader to provide a streaming request body, which doesn’t fit nicely with the <code>io.Writer</code> based
support.</p>
<p>I found a couple of answers solving this problem, either by manually encoding the multipart headers or by running a
separate goroutine to write into a pipe which can then be read by the HTTP client.</p>
<p>I settled on something in between: writing the headers to a byte buffer, then slicing the buffer and composing
an <code>io.MultiReader</code> from the parts and the body reader from the S3 <code>getObject</code> response.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>Having seen Peter Bourgon’s post
on <a href="https://peter.bourgon.org/blog/2019/02/12/multipart-http-responses.html">multipart http responses</a> come up in the
Golang Weekly newsletter, I thought I would write some notes on a related problem I was looking at recently: how to
stream a multipart upload to a remote HTTP server.</p>
<p>My specific use case is to receive a stream from AWS S3, and pipe it into the upload API for Asana. It’s probably ending
up back in S3, but that’s neither here nor there.</p>
<p>The multipart writer provided with Go’s standard library makes serving multipart responses really easy, but sending an
HTTP request needs an io.Reader to provide a streaming request body, which doesn’t fit nicely with the <code>io.Writer</code> based
support.</p>
<p>I found a couple of answers solving this problem, either by manually encoding the multipart headers or by running a
separate goroutine to write into a pipe which can then be read by the HTTP client.</p>
<p>I settled on something in between: writing the headers to a byte buffer, then slicing the buffer and composing
an <code>io.MultiReader</code> from the parts and the body reader from the S3 <code>getObject</code> response.</p>
<pre><code>// Error checking omitted
output, _ := r.s3.GetObject(&#x26;s3.GetObjectInput{
	Bucket: aws.String(bucket),
	Key:    aws.String(key),
})

var contentType string
if output.ContentType != nil {
	contentType = *output.ContentType
} else {
	contentType = "application/octet-stream"
}

// Write header
buffer := &#x26;bytes.Buffer{}
partWriter := multipart.NewWriter(buffer)
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition",
	fmt.Sprintf(`form-data; name="%s"; filename="%s"`,
		escapeQuotes(field), escapeQuotes(filename)))
h.Set("Content-Type", contentType)

// Writes part header to buffer, then
// we capture the size for later
partWriter.CreatePart(h)
headerSize := buffer.Len()

// Write footer (without body)
partWriter.Close()

// Create request
request, _ := http.NewRequest(http.MethodPost, c.getURL(path), io.MultiReader(
	bytes.NewReader(buffer.Bytes()[:headerSize]),
	r,
	bytes.NewReader(buffer.Bytes()[headerSize:])))

request.Header.Add("Content-Type", partWriter.FormDataContentType())
resp, _ := httpClient.Do(request)
</code></pre>
<p>This worked well, and avoids needing to read the whole object into memory or caching it to disk. A cleaner solution
might be to implement a reader-based equivalent in the multipart package.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Generic Adapters in Go]]></title>
            <link>https://kothar.net/blog/generic_adapters</link>
            <pubDate>Sun, 13 Aug 2017 22:49:00 GMT</pubDate>
            <guid>https://kothar.net/blog/generic_adapters</guid>
            <description>
              <![CDATA[<p>While playing about with ideas for generic types in Go, I have come up with what I’m calling a ‘Generic Adapter’.</p>
<p>The approach I’ve taken here is essentialy that described by Kevin Gillette in <a href="https://groups.google.com/d/msg/golang-nuts/hT12Cno8k_E/ycOLrqhrDsYJ">a 2013 Post to golang-nuts</a>. It implements ‘erasure-style’ generics by wrapping the generic implementation of each method with an adapter function which handles type assertions for the caller.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>While playing about with ideas for generic types in Go, I have come up with what I’m calling a ‘Generic Adapter’.</p>
<p>The approach I’ve taken here is essentialy that described by Kevin Gillette in <a href="https://groups.google.com/d/msg/golang-nuts/hT12Cno8k_E/ycOLrqhrDsYJ">a 2013 Post to golang-nuts</a>. It implements ‘erasure-style’ generics by wrapping the generic implementation of each method with an adapter function which handles type assertions for the caller.</p>
<pre><code>type Container struct {
	T SomeType
	
	Set func(value SomeType) SomeType
}

type Int64Container struct {
	T int64
	
	Set func(value int64) int64
}

func myProgram() {
	c := &#x26;Int64Container{}
	generic.Factory(Container{}).Create(c)
}
</code></pre>
<p>Source code implementing a proof of concept can be <a href="https://bitbucket.org/mikehouston/generic" title="mikehouston/generic">found on Bitbucket</a>. I can’t really say I recommend anyone using it, but it’s satisfying to show that it works!</p>
<h2>Background</h2>
<p>There has been a debate raging in the Go community about <a href="https://github.com/golang/go/issues/15292" title="proposal: spec: generic programming facilities">support for generic types</a> since the language was first released.</p>
<p>Recently this has been rekindled by the announcement that the Go team would be looking for case studies to support the inclusion of new (potentially V1-incompatible) features in Go 2. They are currently <a href="https://github.com/golang/go/wiki/ExperienceReports#generics" title="ExperienceReports">collecting experience reports</a> for why people would find generics useful in Go.</p>
<p><a href="https://appliedgo.net/generics/" title="Who needs generics?">Many solutions</a> have been suggested to add support within Go 1, the most pragmatic of which is <a href="https://github.com/cheekybits/genny" title="Genny">generating new code</a> from generic templates. This provides all of the benefits of generics without needing any additional support in the language. The disadvantage is that there’s an extra pre-compilation step to process the templates.</p>
<h2>Why do people want generics?</h2>
<h3>Type safety</h3>
<p>The canonical example of a generic type is probably a linked list. Go provides built-in generic slice and map types, but if you need something like a linked list or an ordered map, you need to implement them yourself.</p>
<p>Go provides a few <a href="https://golang.org/pkg/container/" title="Directory /src/container">container data structures</a>, but they all operate on <code>interface{}</code> types. This means that you must include type assertions when you are getting elements out of the data structure.</p>
<p>Also, since the compiler can’t check which types should be added to the container, you (or a user of your code) could add arbitrary items without any compile or runtime warning until the item is retrieved and fails your type assertion.</p>
<h3>Type documentation</h3>
<p>If you pass an abstract container to other code, there is no attached information about which types you are using. This means the type must be noted in your documentation and kept up to date, but there’s no automatic check that this information is correct.</p>
<h3>Libraries</h3>
<p>If you are <a href="https://www.reddit.com/r/golang/comments/6c87mb/code_generator_for_making_genericlike_types/dhu3h47/" title="Comment by allowthere about generic code generators">writing a data structure library</a>, you don’t necessarily know the types that will be used ahead of time, and so can’t generate specialised implementations.</p>
<p>This <a href="https://www.reddit.com/r/golang/comments/6c87mb/code_generator_for_making_genericlike_types/dhu3h47/" title="Comment by allowthere about generic code generators">comment by allowthere</a> about code generation on Reddit captures the issue:</p>
<blockquote>
<p>The thing is that this won’t help the one place generics are actually really useful - libraries.<br>
You don’t need generics when writing your own code, and when you do, you can always cut and paste+grep.<br>
The real need for generics is libraries. Let’s say I want to make a “better” channel library. Let’s say I want to make a growable channel.<br>
Sure, I can make it for myself, and make a StringChan, IntChan, AcmeTypeChan, etc. But what if I want to share it?<br>
I have to make an InterfaceChan.<br>
This is where I would need generics, and where code-gen won’t help.</p>
</blockquote>
<h3>Performance</h3>
<p>Performance, and to a smaller degree memory usage, are degraded by requiring data structures to use interfaces instead of concrete types. They add an extra level of indirection which can have an impact if the data structure is in a critical part of the code. Function calls can be inlined, but type assertions still need to be made when retrieving abstract types.</p>
<p>A few quick benchmarks show that setting a primitive <code>int</code> value into a slice of <code>[]interface{}</code> is slower than a directly typed slice of <code>[]int</code>, but the overhead for reading back that value with a type assertion is small:</p>
<pre><code>BenchmarkInterfaceSliceSet-8              50000000               25.3 ns/op
BenchmarkPrimitiveSliceSet-8            2000000000               0.35 ns/op

BenchmarkInterfaceSliceGet-8            2000000000               0.47 ns/op
BenchmarkPrimitiveSliceGet-8            2000000000               0.30 ns/op
</code></pre>
<p>In practice, if a data structure is in an inner loop of a heavily used function, it’s probably advisable to cut out an abstract data structure altogether and use a raw slice or map if possible. On the other hand, the Go compiler is pretty good at optimising and inlining calls to these structures.</p>
<h2>Generic Adapters</h2>
<p>I’m sure this concept must have another name already, but I can’t find one yet. In essence, you can take a generic interface and create a type-safe wrapper which adapts typed functions to their generic counterparts.</p>
<h3>Approach</h3>
<p>There are two steps that the implementer of the generic type completes:</p>
<ul>
<li>Define a generic type using func fields</li>
<li>Create a generic implementation and register it with the adapter library</li>
</ul>
<p>There are additionally two steps required to use a specialised version of that type:</p>
<ul>
<li>Create a struct matching the generic type, but with func fields specialised to the type you wan to use.</li>
<li>Use the adapter library to initialise an instance of the type</li>
</ul>
<h3>Func fields?</h3>
<p>One limitation of this approach is that we can’t use interfaces directly, but must use function fields in a struct instead. The reason for this is that the Go reflection package doesn’t allow us to dynamically create implementations for interface types, but does allow us to create functions conforming to a given signature, and assign them to a field.</p>
<p>There is an <a href="https://github.com/golang/go/issues/16522" title="reflect: NamedOf">open issue for this</a> on GitHub, currently tagged for Go 1.10, so this may eventually be possible with just interfaces, which would be much nicer, as you could then use the generated wrapper with compatible interfaces.</p>
<h3>Simple example</h3>
<p>Here is a simple List interface, with a single generic parameter <code>T</code>. We aren’t claiming anything about the implementation yet:</p>
<pre><code>type ListItem interface{}

type List struct {
	T ListItem

	Len    func() int
	Get    func(i int) ListItem
	Set    func(i int, item ListItem) ListItem
	Append func(item ListItem)
	Insert func(i int, item ListItem)
	Remove func(i int) ListItem
}
</code></pre>
<p>In fact, it is implemented using a slice. This is probably not a useful abstraction, but it illustrates our point:</p>
<pre><code>type ListImpl struct {
	T     ListItem
	items []ListItem
}

func newList() interface{} {
	return &#x26;ListImpl{}
}

func (l *ListImpl) Len() int {
	return len(l.items)
}

func (l *ListImpl) Get(i int) ListItem {
	return l.items[i]
}

func (l *ListImpl) Set(i int, item ListItem) ListItem {
	existing := l.items[i]
	l.items[i] = item
	return existing
}

func (l *ListImpl) Append(item ListItem) {
	l.items = append(l.items, item)
}

func (l *ListImpl) Insert(i int, item ListItem) {
	l.items = append(l.items[:i], append([]ListItem{item}, l.items[i:]...)...)
}

func (l *ListImpl) Remove(i int) ListItem {
	existing := l.items[i]
	l.items = append(l.items[:i], l.items[i+1:]...)
	return existing
}
</code></pre>
<p>In this implementation, we are stating that we accept any specialisation of <code>T</code> which conforms to <code>ListItem</code></p>
<p>Once we have these two types, we can register the implementation with the adapter library:</p>
<pre><code>func init() {
	generic.Factory(List{}).
		Register(ListImpl{}, newList)
}
</code></pre>
<p>This is saying that <code>ListImpl</code> can be used as a generic implementation of <code>List</code>.</p>
<p>Now we can create a specialised type with the appropriate type assertions:</p>
<pre><code>type Int64List struct {
	T int64
	
	Len    func() int
	Get    func(i int) int64
	Set    func(i int, item int64) int64
	Append func(item int64)
	Insert func(i int, item int64)
	Remove func(i int) int64
}
</code></pre>
<p>Notice that all occurrences of <code>ListItem</code> in the original <code>List</code> type have been replaced by <code>int64</code>. The type parameter <code>T</code> is also changed to indicate its type.</p>
<p>Finally, we create a new instance of the specialised type:</p>
<pre><code>func NewInt64List() *Int64List {
	l := &#x26;Int64List{}
	generic.Factory(List{}).Create(l)
	return l
}
</code></pre>
<p>Calling the <code>Create</code> method on the factory will instantiate a new instance of the implementation and map its methods into the func fields on <code>l</code>.</p>
<h3>Specialised implementations</h3>
<p>So, now we have the generic adapter factory (I’m sure there are a few more design patterns that could be thrown in there), we can use the generic implementation with type-safe wrappers.</p>
<p>This addresses the first few use cases listed above: type safety, documentation and libraries.</p>
<p>However, if we want to address the performance issue, we still need a bit more. Go can’t dynamically generate new code at runtime, so it can’t optimise the generic implementation with knowledge of the concrete types.</p>
<p>Here’s where we can fall back on existing approaches: we can generate a specialised implementation using the concrete types before compilation, and register these as implementations of the generic types.</p>
<p>In this way we are making use of the factory pattern, providing the most appropriate implementation for the given situation.</p>
<p>Here’s an example of registering a specialised <code>string</code> list:</p>
<pre><code>type StringListImpl struct {
	T     string
	items []string
}

// Type-specifc methods ...

func init() {
	generic.Factory(List{}).
		Register(StringListImpl{}, newStringList)
}
</code></pre>
<h3>Performance</h3>
<p>To be honest, it’s a bit slow:</p>
<pre><code>BenchmarkDirectListCreate-8             2000000000               1.81 ns/op
BenchmarkGenericListCreate-8             1000000              1475 ns/op
BenchmarkSpecialisedListCreate-8         1000000              1734 ns/op

BenchmarkDirectListAppend-8             200000000                6.36 ns/op
BenchmarkGenericListAppend-8             2000000               739 ns/op
BenchmarkSpecialisedListAppend-8        10000000               164 ns/op

BenchmarkDirectListSet-8                2000000000               0.53 ns/op
BenchmarkGenericListSet-8                2000000               674 ns/op
BenchmarkSpecialisedListSet-8           10000000               170 ns/op

BenchmarkDirectListGet-8                2000000000               0.61 ns/op
BenchmarkGenericListGet-8                3000000               443 ns/op
BenchmarkSpecialisedListGet-8           10000000               172 ns/op
</code></pre>
<p>That’s not exactly a surprise, due to the use of reflection in the new method shims.</p>
<pre><code>BenchmarkReflectTypeOf-8                300000000                5.51 ns/op
BenchmarkReflectValueOf-8               30000000                53.6 ns/op
</code></pre>
<p><code>DirectList*</code> and <code>SpecialisedList*</code> are actually using the same underlying implementation, so it’s clear how much overhead this approach introduces. If the data structure you are adapting is a bit meatier, then this overhead may still be acceptable. Similarly if the data structure is not used in a performance-critical section of code.</p>
<p>When the function signature of the implementation matches the wrapper, we avoid as much reflection as possible by directly mapping to the implementation function, but there’s still an extra ~170ns/op. At first I thought this was because the function can’t be inlined, but this shouldn’t cause such a large overhead: calls through interface types can’t be inlined either, and that only adds ~2.5ns:</p>
<pre><code>BenchmarkDirectListSet-8                2000000000               0.53 ns/op
BenchmarkInterfaceListSet-8             500000000                2.98 ns/op
BenchmarkFuncListSet-8                  500000000                3.04 ns/op
BenchmarkReflectFuncListSet-8           10000000               185 ns/op
</code></pre>
<p>I tried a couple of variations, including manually setting the func field to the method pointer from the implementation type, but this doesn’t seem to introduce more overhead than just using an interface.</p>
<p>If I use <code>reflect.Value.Set</code> to assign the method pointer to the func field, we suddenly get 180ns overhead. What’s going on?</p>
<p>It turns out that using reflection to assign a method pointer to a func field is not the same as doing it directly, and results in an extra instance of <code>reflect.methodValueCall</code> being created to wrap the method with a function. I don’t know if there is a better way of doing this. It looks like the compiler is able to generate a more efficient function -> method wrapper than is possible at runtime.</p>
<p>In summary: room for improvement.</p>
<h2>Conclusion</h2>
<p>There’s still plenty missing from the <a href="https://bitbucket.org/mikehouston/generic" title="mikehouston/generic">proof of concept implementation</a>, but hopefully I’ve done the idea justice.</p>
<p>I’ve got some ideas for more extensions to the adapter to allow more complex use cases, but that will have to wait for a future date.</p>
<p>For now, the adapter provides runtime type safety checks to be added to a generic implementation of a data structure. It also allows specialised implementations, such as those generated from a template, to be registered and used where appropriate without changing the code that uses the generic type.</p>
<p>There are some drawbacks to using the adapted wrapper, such as performance and use of func fields, but maybe some people will find the benefits attractive.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Backblaze B2 Performance Part 2]]></title>
            <link>https://kothar.net/blog/b2-part-2</link>
            <pubDate>Tue, 15 Dec 2015 19:18:00 GMT</pubDate>
            <guid>https://kothar.net/blog/b2-part-2</guid>
            <description>
              <![CDATA[<p>After my <a href="/blog/b2-part-1">earlier experiments with B2</a>, I had an extremely interesting call with Backblaze about B2 features and performance.</p>
<p>Firstly, they have recently added a caching layer to speed up serving repeatedly requested files. This reduces the delay as the file is reassembled from Reed-Solomon slices.
They also suggested that I do some new tests, as they thought I should be seeing faster speeds, even for first-access.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>After my <a href="/blog/b2-part-1">earlier experiments with B2</a>, I had an extremely interesting call with Backblaze about B2 features and performance.</p>
<p>Firstly, they have recently added a caching layer to speed up serving repeatedly requested files. This reduces the delay as the file is reassembled from Reed-Solomon slices.
They also suggested that I do some new tests, as they thought I should be seeing faster speeds, even for first-access.</p>
<p>I ran some new tests, but I was still getting lacklustre download performance. Some streams dowloaded at &#x3C;150 KiB/s, with the fastest reaching 1.6MiB/s.
This is much better than the last set of tests I ran, but still slower than Backblaze thought I should be getting (25-50MiB/sec).</p>
<p><em>Speed tests - PlusNet UK Domestic Broadband, 40:20 Mbit/sec:</em></p>
<pre><code>Upload      Sequential:Min Sequential:Max Sequential:Average Parallel:Min Parallel:Max Parallel:Average Parallel:Total
1.3 MiB/sec 0.1 MiB/sec    4.1 MiB/sec    2.1 MiB/sec        0.1 MiB/sec  1.3 MiB/sec  1.0 MiB/sec      0.4 MiB/sec
</code></pre>
<p>PlusNet are known for their traffic shaping, so I decided to try the same test from the cloud. In this case, an Amazon EC2 instance in Ireland.</p>
<p><em>Speed tests - EC2 t2.medium, EU-West-1:</em></p>
<pre><code>Upload      Sequential:Min Sequential:Max Sequential:Average Parallel:Min Parallel:Max Parallel:Average Parallel:Total
3.3 MiB/sec 1.6 MiB/sec    5.3 MiB/sec    2.9 MiB/sec        1.2 MiB/sec  1.9 MiB/sec  1.5 MiB/sec      4.6 MiB/sec
</code></pre>
<p>That’s a bit better, but the rage from minimum to maximum still seems quite large.
t2.medium instances have low-performance networking, but I would still expect them to be able to handle > 5 MiB/sec.
<a href="http://epamcloud.blogspot.co.uk/2013/03/testing-amazon-ec2-network-speed.html">This source</a> benchmarks a t1.micro instance at around 60Mbit/s if I’m reading the charts correctly. So, suspecting the problem may be disk IO rather than the network, I added a mode to my tool which discards the download rather than writing it to disk:</p>
<p><em>Speed tests - EC2 t2.medium, EU-West-1, —discard:</em></p>
<pre><code>Upload      Sequential:Min Sequential:Max Sequential:Average Parallel:Min Parallel:Max Parallel:Average Parallel:Total
N/A         3.1 MiB/sec    6.0 MiB/sec    4.2 MiB/sec        1.3 MiB/sec  1.4 MiB/sec  1.3 MiB/sec      4.8 MiB/sec
</code></pre>
<p>A little better again, but not much.
Finally, I tried a DigitalOcean droplet in San Francisco. This is as close to BackBlaze as I could get.</p>
<p><em>Speed tests - Digital Ocean SFO1:</em></p>
<pre><code>Upload      Sequential:Min Sequential:Max Sequential:Average Parallel:Min Parallel:Max Parallel:Average Parallel:Total
1.8 MiB/sec 2.4 MiB/sec    49 MiB/sec     33.9 MiB/sec       1.5 MiB/sec  42 MiB/sec   21.9 MiB/sec     5.9 MiB/sec
</code></pre>
<p>So, the speed is definitely there, I’m just not close enough.</p>
<p>Finally, although it feels a bit like a sledgehammer, I spun up an Amazon EC2 m4.xlarge instance (“high” network performance) in US-West-1:</p>
<pre><code>$ b2 get -v -j1 --discard riscos-2012-11-01-RC6.zip riscos-2012-11-01-RC6.zip riscos-2012-11-01-RC6.zip riscos-2012-11-01-RC6.zip
riscos-2012-11-01-RC6.zip                              98 MiB [==================] 2.1 MiB/sec 100%
riscos-2012-11-01-RC6.zip                              98 MiB [==================] 51 MiB/sec 100%
riscos-2012-11-01-RC6.zip                              98 MiB [==================] 2.1 MiB/sec 100%
riscos-2012-11-01-RC6.zip                              98 MiB [=================>] 50 MiB/sec 100%
</code></pre>
<h2>Final thoughts</h2>
<p>In summary, the maximum speeds achievable seem to be around 3MiB/sec upload and 50MiB/sec download, but these speeds depend on being close to the Backblaze servers (this applies equally to any cloud service, and is one reason CDNs exist). It should be possible to get more throughput by running multiple streams in parallel.</p>
<p>Downloads seem to start in either a ‘fast’ or ‘slow’ mode, which presumably corresponds to cached vs non-cached access, but which speed you get doesn’t seem to be predictable, at least in my very limited testing.</p>
<p>This is certainly fast enough for home backup use now, and I can only imagine speeds will improve as systems are optimised. If you want to serve files from B2, it seems like a good idea to use some kind of caching layer or CDN until Backblaze have something in place to speed up international transfers.</p>
<hr>
<p>All tests were conducted with the same 98MiB file:  riscos-2012-11-01-RC6.zip</p>
<p>Sequential and parallel downloads were performed 4 times and with 4 simultaneous downloads respectively.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Backblaze B2 and Go]]></title>
            <link>https://kothar.net/blog/b2-part-1</link>
            <pubDate>Mon, 30 Nov 2015 10:54:00 GMT</pubDate>
            <guid>https://kothar.net/blog/b2-part-1</guid>
            <description>
              <![CDATA[<p>I’ve been implementing a Go client library for the Backblaze B2 cloud storage service: <a href="http://github.com/kothar/go-backblaze">go-backblaze</a>.
It works extremely well, but is a bit slow to get files back.</p>
<p><strong>Update 12th December: I’ve spoken to Backblaze, who have been working to improve  performance.  I have performed some new  tests and <a href="/blog/b2-part-2">written them up</a>.</strong></p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>I’ve been implementing a Go client library for the Backblaze B2 cloud storage service: <a href="http://github.com/kothar/go-backblaze">go-backblaze</a>.
It works extremely well, but is a bit slow to get files back.</p>
<p><strong>Update 12th December: I’ve spoken to Backblaze, who have been working to improve  performance.  I have performed some new  tests and <a href="/blog/b2-part-2">written them up</a>.</strong></p>
<p>I’ve implemented a parallel download feature which you can use to download several files at the same time - this doesn’t seem to affect the speed of individual downlaods
very much, so I assume that the downloads are limited for one of two reasons</p>
<ol>
<li>There is a download speed cap in place</li>
<li>The downloads each come from separate parts of the cluster file system, and so don’t affect each other.</li>
</ol>
<p>Downloading 5 copies of the same file in parallel doesn’t seem to affect the download speed, nor do 5 sequential downloads. Whatever you do, each download
seems to run at about 200KiB/sec.</p>
<hr>
<p><em>Note - the <code>b2</code> command used here is my own implementation of a b2 client, rather than the python b2 client provided by Backblaze</em></p>
<p>Uploading 221MiB (sequentially): 1.65MiB/sec</p>
<pre><code>$ time b2 put -v VirtualBox-5.0.10-104061-OSX.dmg N1.dmg Bitcasa_1.5.1615.dmg  Wireshark\ 2.0.0\ Intel\ 64.dmg
VirtualBox-5.0.10-104061-OSX.dmg                     90044871 [==================] 1973.19 KB/s 100%
N1.dmg                                               76457433 [==================] 2142.65 KB/s 100%
Bitcasa_1.5.1615.dmg                                 33741540 [==================] 2380.39 KB/s 100%
Wireshark 2.0.0 Intel 64.dmg                         31522805 [==================] 1838.25 KB/s 100%

real	2m14.334s
user	0m8.893s
sys	0m7.747s
</code></pre>
<p>Downloading sequentially: very slow!</p>
<pre><code>$ time b2 get -v -j 1 VirtualBox-5.0.10-104061-OSX.dmg N1.dmg Bitcasa_1.5.1615.dmg  Wireshark\ 2.0.0\ Intel\ 64.dmg
VirtualBox-5.0.10-104061-OSX.dmg                     90044871 [========>---------]  270.50 KB/s  54%
^C
real	3m1.831s
user	0m7.825s
sys	0m9.693s
</code></pre>
<p>Downloading in parallel: better - 0.56MiB/sec average</p>
<pre><code>$ time b2 get -v -j 5 VirtualBox-5.0.10-104061-OSX.dmg N1.dmg Bitcasa_1.5.1615.dmg  Wireshark\ 2.0.0\ Intel\ 64.dmg
VirtualBox-5.0.10-104061-OSX.dmg                     90044871 [==================]  224.20 KB/s 100%
N1.dmg                                               76457433 [==================]  236.36 KB/s 100%
Bitcasa_1.5.1615.dmg                                 33741540 [==================]  212.05 KB/s 100%
Wireshark+2.0.0+Intel+64.dmg                         31522805 [==================]  213.37 KB/s 100%

real	6m35.563s
user	0m30.205s
sys	0m27.586s
</code></pre>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Organise Asana Projects]]></title>
            <link>https://kothar.net/blog/organise_asana</link>
            <pubDate>Wed, 25 Sep 2013 22:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/organise_asana</guid>
            <description>
              <![CDATA[<blockquote>
<p><strong>UPDATE</strong>: <a href="https://ditto.kothar.net">Ditto</a> - the successor to this tool - <a href="/ditto">has now been launched</a>. Please give it a go and let me know how you get on!</p>
</blockquote>
<p>I was faced recently with wanting to move Asana projects from one workspace to another. Unfortunately this isn’t a feature in the Asana interface (yet).</p>
<p>There’s a PHP script <a href="https://gist.github.com/AWeg/5814427">floating around GitHub</a> which will copy tasks between projects. This requires you to do some manual checking of project IDs, and you need to create the target project in advance, so I’ve extended the script and written a front-end web interface for it.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<blockquote>
<p><strong>UPDATE</strong>: <a href="https://ditto.kothar.net">Ditto</a> - the successor to this tool - <a href="/ditto">has now been launched</a>. Please give it a go and let me know how you get on!</p>
</blockquote>
<p>I was faced recently with wanting to move Asana projects from one workspace to another. Unfortunately this isn’t a feature in the Asana interface (yet).</p>
<p>There’s a PHP script <a href="https://gist.github.com/AWeg/5814427">floating around GitHub</a> which will copy tasks between projects. This requires you to do some manual checking of project IDs, and you need to create the target project in advance, so I’ve extended the script and written a front-end web interface for it.</p>
<p>You can get the <a href="https://bitbucket.org/mikehouston/organiseasana">source code on BitBucket</a>.</p>
<p><strong>Visit</strong> <a href="http://asana.kothar.net"><strong>asana.kothar.net</strong></a> <strong>to use the tool</strong></p>
<p>Before you start, you may need to set up a few things that the tool can’t do on its own though the API.</p>
<ol>
<li>Add your new user account (for your destination workspace) to your old workspace, if not already using the same email address</li>
<li>Log in to Asana with your new account</li>
<li>Add any accounts you want to keep task assignments for to the new workspace</li>
<li>Add any custom fields you want to keep to the new workspace (pro workspaces)</li>
<li>Log in to the tool: <a href="https://asana.kothar.net">https://asana.kothar.net</a></li>
<li>Choose the source workspace and project you’d like to copy</li>
<li>Choose where you’d like to copy to</li>
<li>Hit Go!</li>
</ol>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[File sync and cloud storage client comparison]]></title>
            <link>https://kothar.net/blog/file_sync</link>
            <pubDate>Thu, 25 Jul 2013 22:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/file_sync</guid>
            <description>
              <![CDATA[<p>I’ve been trying out a number of file sync clients recently, finding
that many are trying to solve the same problem and meeting with
varying success. Probably the best-known service in this category
is <a href="https://dropbox.com/">Dropbox</a>, but nearly every big cloud
services company, and a multitude of startups which base their
services on top of them, are providing a ‘cloud drive’ of some
description.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>I’ve been trying out a number of file sync clients recently, finding
that many are trying to solve the same problem and meeting with
varying success. Probably the best-known service in this category
is <a href="https://dropbox.com/">Dropbox</a>, but nearly every big cloud
services company, and a multitude of startups which base their
services on top of them, are providing a ‘cloud drive’ of some
description.</p>
<p>First off, I don’t want to reinvent the wheel here, so you might be
interested in some background reading on cloud storage:</p>
<ul>
<li><a href="https://gizmodo.com/5990589/the-best-way-to-store-your-stuff-in-the-cloud">The Best Cloud Storage Service</a> -
Gizmodo’s roundup of cloud storage in March 2013</li>
<li><a href="https://www.wired.com/insights/2011/10/backblazes-basic-cloud/">Why is BackBlaze so cheap?</a> -
Backblaze’s Basic Cloud Storage is 25 Times Cheaper than Amazon S3</li>
<li><a href="http://www.softwareadvice.com/labs/box-vs-dropbox/">Box vs. Dropbox</a> -
an exhaustive feature comparison of these two services, which may
help understand what I’m going to cover below.</li>
</ul>
<p>The following table breaks down some of the services I’ve tried out,
which hopefully gives you an idea of the range on offer. Further down
I examine some of the issues that you might want to consider when
choosing a cloud storage provider. Even though I’m in the UK, I’ve
used USD for pricing, since most services offer this currency.</p>
<p>If you would like to calculate how much various cloud storage
providers would cost you for a set amount of data storage, you might
like to check
out <a href="http://www.cloudpricing.co.uk/">CloudPricing.co.uk</a>.</p>
<p>| Service | Free storage | Small | $/GB/mo | Large | $/GB/mo | Linking | Shared sync | Versions | Online access | Microbilling |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| <a href="https://www.dropbox.com">Dropbox</a> | 2GB - 18GB | $9.99 / 100GB | 0.1 | $49.99 / 500GB | 0.1 | Yes | Yes | Yes | Yes | No |
| <a href="https://www.backblaze.com/cloud-backup/personal#af6870">Backblaze</a> | - | $5 / Unlimited | - | - | - | No | No | Yes | Yes | No |
| <a href="http://www.amazon.com/gp/feature.html?ie=UTF8&#x26;docId=1000796931">Cloud Drive</a> (Amazon) | 5GB | $0.50* / 20GB | 0.025 | $26.67* / 1TB | 0.027 | Yes | No | Deleted | Yes | No |
| <a href="https://drive.google.com">Google Drive</a> † | 15GB | $4.99 / 100GB | 0.05 | $799.99 / 16TB | 0.05 | Yes | Yes? | Yes | Yes | No |
| <a href="http://l.bitcasa.com/Af9fNHcC">Bitcasa</a> | 10GB | $11.68 / 1TB | 0.012 | - | - | Yes | No | Yes | Yes | No |
| <a href="http://www.adobe.com/products/creativecloud/tools-and-services.html">Creative Cloud</a> (Adobe) ‡ | 2GB | $49.99* / 20GB | ? | - | - | Yes | Yes | Yes | ? | No |
| <a href="https://skydrive.live.com">SkyDrive</a> (Microsoft) | 7GB | $10 / 20GB | 0.5 | $50 / 100GB | 0.5 | Yes | Yes? | Yes | Yes | No |
| <a href="https://www.icloud.com/">iCloud</a> (Apple) † | 5GB | $1.67* / 10GB | 0.167 | $8.33* / 50GB | 0.167 | Photos | No | No? | Yes | No |
| <a href="http://www.livedrive.com/">Livedrive</a> | - | $12+* / 2TB | 0.006 | - | - | Yes | No | No? | Yes | No |
| <a href="https://www.sugarsync.com/">SugarSync</a> | - | $6.25* / 60GB | 0.1 | $33.25* / 500GB | 0.067 | Yes | Yes | Yes | Yes | No |
| <strong>Bring your own storage</strong> | | | | | | | | | | |
| <a href="http://labs.bittorrent.com/experiments/sync.html">BitTorrent Sync</a> | - | - | - | - | - | Yes | Yes | - | - | No |
| <a href="http://sparkleshare.org/">Sparkleshare</a> | - | - | - | - | - | No | Yes | Yes | Yes | - |
| <a href="http://en.wikipedia.org/wiki/Rsync">rsync</a> | - | - | - | - | - | No | No | - | - | - |
| <a href="https://s3sync.net/wiki.html">s3sync</a> + <a href="http://aws.amazon.com/s3/">Amazon S3</a> | - | &#x3C;1TB | 0.095 | >5PB | 0.055 | Yes | No | Yes | Yes | Yes |</p>
<p>* Annual subscription - † Shared with other services - ‡ Includes
other services - <strong>Prices and storage were current on July 26th 2013</strong></p>
<h2>Usage tiers and microbilling</h2>
<p>When I first read about <a href="https://www.dropbox.com/">Dropbox</a>, I
thought: ‘100GB for only $10 a month - how can they do that?!‘. As you
can see from the table, they are not even the cheapest service, and
come in only a hair above <a href="https://aws.amazon.com/s3/">Amazon’s S3</a> (
Simple Storage Service) on a per-GB rate. Given that they use S3 for
their underlying storage, how do Dropbox make any money? And they give
away loads of free accounts too!</p>
<p>The answer should be obvious to people who are familiar with the way
Dropbox works. When you share a folder, the space used counts against
both user accounts, essentially saving Dropbox storage space. The more
people sharing a folder, the less space Dropbox actually needs to be
able to provide overall. In addition, due to the way the storage tiers
work, you have to pay for the entire amount, even if you are only
using some of it. Say you have filled your 2GB free account, and
upgrade to 100GB. Now, until you store the full 100GB, you’re not
getting the actual $0.10 per GB rate. In fact, when you’re storing
50GB of data, you’re paying $0.20/GB, and some of that will most
likely be shared too.</p>
<p><a href="https://www.amazon.com/gp/feature.html?ie=UTF8&#x26;docId=1000796931">Amazon CloudDrive</a>
is another service which is much cheaper per GB than the base storage
costs would suggest. OK, so Amazon operate S3, so they get better
rates, but they also include MP3 streaming as a large part of their
service. MP3s you purchase from Amazon don’t count against the storage
space, but in theory, if lots of people upload identical DRM-free MP3
files, Amazon would only need to store one copy of the file; I don’t
know if they actually apply that kind of process though.</p>
<p>So, basically, most users are going to end up paying relatively high
average unit costs for the amount of data they store. Maybe some users
will fill their allotment, but then nobody likes to live that close to
the edge - what if they need to share another file, and their account
is full? My conclusion is that services which offer relatively small
increments (say 10GB at a time), rather than massive ‘Pro’ usage tiers
which you won’t fill for a long time, are probably better value in the
end.</p>
<h2>Unlimited storage</h2>
<p>Some services offer ‘Unlimited’ storage. Even if prices are calculated
as if they were providing 1TB of storage, these services still look
cheap.</p>
<p>Out of the two I looked at
here, <a href="https://www.backblaze.com/cloud-backup/personal#af6870">Backblaze</a>
is pitched as a pure backup service. It doesn’t offer live sync,
linking or sharing; only one-way backup and restore. They can get
their costs down because they don’t need to have fast access to all
their data all the time - they can optimise for cost above all other
considerations.</p>
<p>This has some similarities to
the <a href="https://aws.amazon.com/glacier/">Amazon Glacier</a> service, which
is the long-term archival version of S3. It has costs starting at
$0.01/GB, which would place it in the realm of a $10/TB/month storage
service. Backblaze is still cheaper. As with Dropbox, I suspect they
can optimise here by assuming that most users actually only want to
store their files, not the entire contents of their hard drives. In my
case, my Backblaze data set is just over 100GB, and assuming that some
compression is also used to reduce the final size on the server, I can
imagine the costs for Backblaze coming in at around $1/month. If they
charge me $5, they can afford to support a few users who want to back
up terabytes of data.</p>
<p>I assume similar ecomomies are made with the other ‘Unlimited’
provider listed here, <a href="http://l.bitcasa.com/Af9fNHcC">Bitcasa</a>. They
aren’t pitching themselves as ‘just’ a backup service, though they do
make mention of using their service for backing up data. It costs a
bit more per month, but you get live access to your data, and two-way
sync.</p>
<h2>Data Security</h2>
<p>Is your data secure on these cloud services? Do you trust them not to
go bust or lose the data, and do you trust them not to let hackers get
into their systems and rummage through your files? What if you max out
your credit card and a payment bounces? Paranoia will only take you so
far, and in the end the data centres where these companies have their
servers are probably more secure than your home. That’s not to say you
can’t take precautions.</p>
<p>Keeping an offline backup in addition to an online backup is probably
wise. If nothing else, it will be quicker to restore if you need it.
Backblaze will send you up to 3TB of files from your archive on an
external hard drive, but they have to still have your data.</p>
<p>For ultimate security, you can supply your own encryption key. This is
supported by some of these services, but not all. The only one that I
noticed advertising support was Backblaze. This of course has the
tradeoff that you have to store the key, possibly on a USB drive or SD
card, and put it somewhere safe. If you lose it, your backup is
useless. Maybe you could tattoo it onto your arm.</p>
<p><img src="/static/encryption.png" alt="Backblaze encryption settings dialog"></p>
<h2>Bring your own storage</h2>
<p>A few pieces of software exist which can help you recreate some of the
features of these services on your own server, or using other data
storage services with pay-as-you-go pricing, such as Amazon S3 and
Glacier.</p>
<p>One of the most basic ways to transfer files is using an FTP or SFTP
client. This can be automated, but synchronising lots of files this
way can be very inefficient and error prone. For this reason, tools
like <a href="https://en.wikipedia.org/wiki/Rsync">rsync</a>
and <a href="https://www.cis.upenn.edu/~bcpierce/unison/">unison</a> are
generally used for this purpose instead.</p>
<p>But where are you going to transfer your files to? If you want to use
these tools you will need a server somewhere with which to
synchronise. Hard drives currently cost around
$0.045/GB <a href="https://pcpartpicker.com/parts/internal-hard-drive/#sort=a6">at least $0.034/GB</a>
based on $100 for a 3TB drive (updated March 2014), and are warrantied
for between 2 and 3 years. If you assume that you will get at least 2
years out of them, run them as a mirrored pair for reliability, and
you can find a low-powered NAS unit to run as a server for around
$100, which
costs <a href="http://www.citytrf.net/costs_calculator.htm">about $75 a year to run</a> (
a lot of estimates here), you can estimate a cost for storage to
$450/2 years, or $18.75/month for ~3TB. This works out as
$0.00625/GB/month. Not bad, if you need the majority of the 3TB.</p>
<p>You’d still need to put this mini-server somewhere, and ideally it
would be off-site. There’s a lot to consider, and it takes time to set
up and configure. On the other hand, if you take out a basic hosting
package you can expect to pay around $1/GB/month for storage,
depending on the hosting provider. Some services for web hosting might
offer ‘unlimited’ storage, but they generally state that this doesn’t
cover use for private file storage and backup.</p>
<p>So if none of these approaches seem attractive, and you don’t already
have a server with some spare storage, what other options are there?
The obvious thing to do is to look at how services like Dropbox store
their data, and cut out the middleman. In this case, we go to Amazon
S3.</p>
<p>S3 is a much lower-level storage service than the consumer-targeted
sync clients. It’s aimed mainly at developers to use as the backing
store for their data-intensive apps and websites, and until recently
has had very few user-friendly tools. <a href="http://s3sync.net/wiki">s3sync</a>
is a tool designed to behave in the same way as rsync, but using S3 as
a remote store instead of a standard file system. Once you have the
files on S3, the web interface is very good, and allows you to browse
the files and folders as you would expect.</p>
<p>If you can find tools you are happy with, S3 has one huge advantage,
assuming you aren’t close to the upper end of one of the usage tiers
on a subsidised service: it’s PAYG. This means if you’re storing 12GB,
you pay $1.14, rather than the $10 you would need to pay Dropbox for
anything over the free allowance. On the downside, you lose most of
the cool features Dropbox provides. It’s possible to configure the S3
permissions in similar ways to Dropbox, including sending ‘authorized’
links to people (see below), but it’s nowhere near as user-friendly.</p>
<h2>Sharing links</h2>
<p>The ability to share files is essentially a form of social networking,
and has driven adoption of many sharing services
like <a href="https://www.wetransfer.com/">weTrasfer</a> - it’s free
word-of-mouth marketing. Dropbox has one of the best implementations
of this feature.</p>
<p>If you are providing your own storage, can you give someone a link to
a file you want to share? In general, yes, but you need to set up your
server to provide external access to those files. At its simplest,
this could just be a web site with a directory index, maybe with a
password protecting it. But there’s no limit really to how
sophisticated you can make such sharing features, depending on how
much time you want to spend setting up a functional file server.</p>
<p>If you are using S3 to host your files, the web interface allows you
to make time-limited links to a single file, or to set permissions
which allow the public or specific Amazon Web Services users to access
the files.</p>
<h2>Peer to peer</h2>
<p>Finally, a new service which I read about only recently
is <a href="http://labs.bittorrent.com/experiments/sync.html">BitTorrent Sync</a>.
If you only need the folder sync aspect of one of these file sharing
services, and can rely on the number of copies on different systems
being sufficient to safeguard the files, then this might be a good
solution. It allows all the systems which wish to share files with
each other to talk directly (most of the time) and swap the files
without ever using a central server. It does rely on peers being
online at the same time to swap files though. Of course, you could
still use a copy of BTSync on your server to keep a public copy as
described above, and this would also mean that a copy would always be
available for peers to sync with.</p>
<p>BTSync even provides a way to give people a one-way sync of a folder,
which is a bit like sending them a share link, although they would
need to install the client software to allow them to receive the
files.</p>
<h2>Conclusions</h2>
<p>After all that, I’m not sure I really have much in the way of a
conclusion. Peer-to-peer sharing with BTSync seems to be the easiest
way to get file sync working without needing your own server or paying
for anything, and provides some features for sharing links with other
people and collaborating on shared folders. On the down side, it
doesn’t provide a simple web interface to download a shared file, and
you would still need to set up a server to handle those kinds of
features.</p>
<p>Out of the paid services, the best option depends on what you need.</p>
<p>If you want a backup
service, <a href="https://www.backblaze.com/cloud-backup/personal#af6870">Backblaze</a>
seems like the best option at the moment.<br>
If you want a folder sync client for collaborative
work, <a href="https://dropbox.com/">Dropbox</a> is hard to beat, but
expensive. <a href="http://www.bittorrent.com/sync">BitTorrent Sync</a> is very
close, but missing the easy file sharing features which make Dropbox
so popular. <a href="http://sparkleshare.org/">Sparkleshare</a> seems like it
could compete for collaboration, but requires server setup.<br>
And if you just want to share files with
people, <a href="http://l.bitcasa.com/Af9fNHcC">Bitcasa</a> is probably the best
value, as its unlimited plan removes file size from consideration.</p>
<p>Of course, there’s nothing to stop you using more than one service,
perhaps even with overlapping synchronisation folders. For example,
you could use Bitcasa to store all your files on the server, and also
create a BTSync folder inside it which would allow you to retrofit
Bitcasa with the collaboration features it’s missing…</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Plex, SSL and Nginx]]></title>
            <link>https://kothar.net/blog/plex_nginx</link>
            <pubDate>Sun, 24 Mar 2013 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/plex_nginx</guid>
            <description>
              <![CDATA[<p>I’ve been experimenting with the <a href="http://www.plex.tv/">Plex Media Server</a>, and decided to have a go at securing remote access to the ‘Web Client’ interface using SSL (by default it runs over an unencrypted HTTP connection).</p>
<p>This post is essentially a gathering of instructions from where I found them on the internet.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>I’ve been experimenting with the <a href="http://www.plex.tv/">Plex Media Server</a>, and decided to have a go at securing remote access to the ‘Web Client’ interface using SSL (by default it runs over an unencrypted HTTP connection).</p>
<p>This post is essentially a gathering of instructions from where I found them on the internet.</p>
<p>My server is set up as follows:</p>
<ul>
<li><a href="http://www.ubuntu.com/download">Ubuntu</a> 12.04 64-bit</li>
<li>Latest Plex Media Server distribution</li>
<li><a href="http://wiki.nginx.org/Main">Nginx</a> 1.3.14 development build (more on this later)</li>
</ul>
<h2>Installing Plex</h2>
<p>[Nov 2021]: Plex have changed some of their URLs and the wiki no longer seems to be available.</p>
<p>You can download a debian package for Ubuntu from the <a href="https://www.plex.tv/media-server-downloads/#plex-media-server">download page on the new Plex website</a>.</p>
<p>Support can be found on the <a href="https://forums.plex.tv/c/plex-media-server/computers/21">Plex forums</a>.</p>
<p><em>The below information is no longer up to date</em></p>
<hr>
<p>Installing Plex is straightforward - I haven’t deviated from the standard instructions at all.</p>
<p>These are the instructions for <a href="https://web.archive.org/web/20130826143016/http://wiki.plexapp.com/index.php/Downloads#Ubuntu_-_PMS">installing using apt</a>, which will ensure all the necessary dependencies get installed:</p>
<blockquote>
<p>For Ubuntu installation, add this to your <code>/etc/apt/sources.list.d/plex.list</code> file:</p>
<pre><code>deb http://www.plexapp.com/repo lucid main
</code></pre>
<p>Then add the repository key:</p>
<pre><code>$ wget http://plexapp.com/plex_pub_key.pub
$ sudo apt-key add plex_pub_key.pub
$ rm plex_pub_key.pub
</code></pre>
<p>Then install with the following command:</p>
<pre><code>$ sudo apt-get update
$ sudo apt-get install plexmediaserver
</code></pre>
<p>Additional information can be found in the <a href="https://web.archive.org/web/20140406072618/https://forums.plex.tv/index.php/forum/85-linux/">Linux PMS forum</a>.</p>
</blockquote>
<h2>Installing Nginx</h2>
<p>The current stable release of Nginx (1.2.x at the time of writing) doesn’t support the <a href="http://en.wikipedia.org/wiki/WebSocket">WebSocket</a> protocol, which is used in the Plex web interface to update the interface in real-time as the media server updates.</p>
<p>Luckily the latest development versions after 1.3.13 <em>do</em> support WebSocket, but they’re not in the main Ubuntu package repository, for obvious reasons. You can install the latest version from a development package repository by following <a href="https://github.com/mozilla-services/circus/issues/371">these instructions</a>:</p>
<blockquote>
<p>On Ubuntu you can install Nginx>=1.3.13 from Chris Lea’s development branch <a href="https://launchpad.net/~chris-lea/+archive/nginx-devel">PPA</a>, as so,</p>
<pre><code>sudo apt-get install python-software-properties
sudo add-apt-repository ppa:chris-lea/nginx-devel
sudo apt-get update
sudo apt-get install nginx
nginx -v
</code></pre>
</blockquote>
<h2>Configuring Nginx</h2>
<p>I have removed the <code>default</code> configuration from <code>/etc/nginx/sites-enabled</code> and created a new configuration called <code>plex</code>.</p>
<p>The first part of the configuration redirects connections on port 80 to the SSL connection on port 443. This is necessary because if you visit <strong>https</strong>://example.com/web, Plex will redirect you to <strong>http</strong>://example.com/web/index.html.</p>
<pre><code>server {  
 rewrite ^(.*) https://$host$request_uri? permanent;  
}
</code></pre>
<p>This will automatically listen on port 80 on all network interfaces: you don’t need to specify anything else unless you are using other virtual hosts.</p>
<p>The next part of the configuration defines the location of the Plex Media Server. For my setup, it’s on the same system as Nginx:</p>
<pre><code>upstream plex {  
 server 127.0.0.1:32400;  
}
</code></pre>
<p>Finally, the meat of the configuration sets up the SSL-enabled reverse proxy. This means we can map the SSL port (443) to the outside world via the external firewall, and access the Plex server using an encrypted connection.</p>
<pre><code>server {
    listen 443;

    ssl on;
    ssl_certificate /etc/nginx/cert/server.crt;
    ssl_certificate_key /etc/nginx/cert/server.key;

    ## Settings to use client certificates for authentication
    ssl_client_certificate /etc/nginx/cert/Plex.pem;
    ssl_verify_client on;

    access_log  /var/log/nginx/plex.access.log;
    error_log  /var/log/nginx/plex.error.log;

    location / {
     proxy_pass  http://plex;
     proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
     proxy_redirect off;
     proxy_buffering off;
     proxy_set_header        Host            $host;
     proxy_set_header        X-Real-IP       $remote_addr;
     proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

     ## Required for Websockets
     proxy_http_version 1.1;
     proxy_set_header        Upgrade         $http_upgrade;
     proxy_set_header        Connection      "upgrade";
     proxy_read_timeout      36000s;  ## Timeout after 10 hours
   }
}
</code></pre>
<p>This should leave you with a working Plex/Web when you connect to <code>https://plex.example.com/web</code></p>
<p><strong>Warning!</strong> Connecting to the web interface by this method circumvents the Plex login. You need to enable some kind of authentication, otherwise anyone can connect to your server. If you want to use the standard plex login, you’ll have to run the gateway on another system than the server itself and turn on authentication for systems on the local network, as it always allows connections from the same system as far as I can tell.</p>
<h2>Authentication and Certificates</h2>
<p>The SSL configuration requires a server certificate. I’m using a self-signed certificate for this, but you could get a free certificate from <a href="https://www.startssl.com/">StartCom</a> if you want something which won’t trigger a browser warning. There are many guides to <a href="http://www.akadia.com/services/ssh_test_certificate.html">creating self-signed certificates</a> online.</p>
<p>In the above example, I have enabled SSL Client Certificate authentication. This is using a self-generated certificate authority to sign access keys. There is a guide to generating the necessary files <a href="https://gist.github.com/mtigas/952344">here</a>.</p>
<p>As a more common use case, you could instead enable password authentication. Again, there are many guides to <a href="http://www.howtoforge.com/basic-http-authentication-with-nginx">setting up auth_basic for Nginx</a>, so I won’t go into details. You just need to replace the <strong>ssl_client_certificate</strong> and <strong>ssl_verify_client</strong> options in the configuration above.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Resampling for fun and profit]]></title>
            <link>https://kothar.net/blog/rtns2012_resampling</link>
            <pubDate>Wed, 05 Dec 2012 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/rtns2012_resampling</guid>
            <description>
              <![CDATA[<p>I’m a co-author on a paper which was presented at RTNS this year. Sadly I didn’t have time to attend the conference, but it’s nice to have been able to contribute!</p>
<p><a href="/assets/RTNS2012_Resampling.pdf">Re-Sampling for Statistical Timing Analysis of Real-Time Systems</a></p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>I’m a co-author on a paper which was presented at RTNS this year. Sadly I didn’t have time to attend the conference, but it’s nice to have been able to contribute!</p>
<p><a href="/assets/RTNS2012_Resampling.pdf">Re-Sampling for Statistical Timing Analysis of Real-Time Systems</a></p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Dynamic Disqus Instantiation]]></title>
            <link>https://kothar.net/blog/disqus_gwt</link>
            <pubDate>Mon, 27 Aug 2012 22:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/disqus_gwt</guid>
            <description>
              <![CDATA[<p>I needed to use Disqus from a GWT app, so I needed to solve the problem of loading threads on demand as virtual pages in the app were changed.</p>
<p>A small amount of reverse engineering and experimentation led me to construct a utility class (below).</p>
<p>I also posted this answer on <a href="http://stackoverflow.com/a/12168712/37416">Stack Overflow</a>.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>I needed to use Disqus from a GWT app, so I needed to solve the problem of loading threads on demand as virtual pages in the app were changed.</p>
<p>A small amount of reverse engineering and experimentation led me to construct a utility class (below).</p>
<p>I also posted this answer on <a href="http://stackoverflow.com/a/12168712/37416">Stack Overflow</a>.</p>
<p>The main insights are:</p>
<ol>
<li>There is an undocumented global parameter called <code>disqus_container_id</code> which allows you to place the comments wherever you like. If that doesn’t work in some future version, my fallback was going to be to temporarily set the id of the target element to <code>disqus_thread</code>, add the comments and then change it back to the original id.</li>
<li>Since this was being developed for GWT using JSNI, I needed to set the global parameters in the original window context, accessible through <code>$wnd</code>. I changed the default Disqus embed code accordingly. I didn’t realise before that all global variables are in the Window object, but I learned something new.</li>
<li>You can re-use the same container, Disqus seems to clear the contents when you activate it.</li>
<li>This leaves lots of copies of the script tag in the DOM. Maybe it would be a good idea to clean these up too once they’ve been used. Alternatively, I might do some more experiments using the <code>DISQUS.reset</code> method described in other answers.</li>
</ol>
<p>Extracting just the crucial information for someone using JS on its own, this should allow you to stick a Disqus thread anywhere you like:</p>
<pre><code>function loadComments(container_id, shortname, identifier, developer) {
    // CONFIGURATION VARIABLES
    window.disqus_container_id = container_id;
    window.disqus_developer = developer ? 1 : 0;
    window.disqus_shortname = shortname; // required
    if (identifier) window.disqus_identifier = identifier;

    // DON'T EDIT BELOW THIS LINE
    (function() {
       var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
       dsq.src = 'http://' + shortname + '.disqus.com/embed.js';
       (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
    })();
}
</code></pre>
<p>And here is the full GWT utility class. I’ve only implemented the parameters I needed so far.</p>
<pre><code>import com.google.gwt.user.client.Element;
import com.google.gwt.user.client.Random;
import com.google.gwt.user.client.ui.Widget;

public class Disqus {

    public static boolean developer = false;
    public static String shortname;

    public static void showComments(Widget where, String identifier) {
        showComments(where.getElement(), identifier);
    }

    public static void showComments(Element where, String identifier) {
        if (shortname == null)
            throw new IllegalArgumentException(
                      "You must specify the disqus shortname before displaying comments");

        // Store the original id of the target element
        String id = where.getId();
        if (id == null) {
            id = "disqus-" + Integer.toHexString(Random.nextInt());
            where.setId(id);
        }

        // Update the id temporarily
        where.setId("disqus_thread");

        // Load the comments
        loadComments(id, shortname, identifier, developer);
    }

    private static native void loadComments(String container_id, String shortname, String identifier, boolean developer) /*-{
        // CONFIGURATION VARIABLES
        $wnd.disqus_container_id = container_id;
        $wnd.disqus_developer = developer ? 1 : 0;
        $wnd.disqus_shortname = shortname; // required
        if (identifier) $wnd.disqus_identifier = identifier;

        // TODO
        // disqus_url

        // disqus_title

        // disqus_category_id

        // DON'T EDIT BELOW THIS LINE (sorry, I've edited it anyway)
        (function() {
            var dsq = $doc.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
            dsq.src = 'http://' + shortname + '.disqus.com/embed.js';
            ($doc.getElementsByTagName('head')[0] || $doc.getElementsByTagName('body')[0]).appendChild(dsq);
        })();
    }-*/;
}
</code></pre>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Spotify for Linux media keys with DBUS]]></title>
            <link>https://kothar.net/blog/spotify-dbus</link>
            <pubDate>Tue, 10 May 2011 22:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/spotify-dbus</guid>
            <description>
              <![CDATA[<p>Now that Spotify has <a href="http://www.spotify.com/se/blog/archives/2011/04/14/upcoming-changes-to-spotify-free-open/" title="Upcoming changes to Spotify Free/Open">discontinued the free account</a>, and restricted me to an ‘open’ account with only 10 hours of music a month, I’ve ended up paying for an ‘unlimited’ account.</p>
<p>While I’m a little disappointed with this turn of events, it does mean I am now able to use the Linux client. It seems pretty decent so far, but I wanted to map some keys to do things like play/pause and next.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>Now that Spotify has <a href="http://www.spotify.com/se/blog/archives/2011/04/14/upcoming-changes-to-spotify-free-open/" title="Upcoming changes to Spotify Free/Open">discontinued the free account</a>, and restricted me to an ‘open’ account with only 10 hours of music a month, I’ve ended up paying for an ‘unlimited’ account.</p>
<p>While I’m a little disappointed with this turn of events, it does mean I am now able to use the Linux client. It seems pretty decent so far, but I wanted to map some keys to do things like play/pause and next.</p>
<p>Luckily DBUS is supported, so you can create new key mappings in Gnome as desired. I found a fairly concise list of the most useful key bindings <a href="http://lnxscene.org/~jylam/spotify.html" title="Multimedia keys for spotify / linux">on jylam’s page</a>:</p>
<pre><code>"dbus-send --print-reply --dest=org.mpris.MediaPlayer2.spotify /org/mpris/MediaPlayer2 org.mpris.MediaPlayer2.Player.PlayPause" XF86AudioPlay
"dbus-send --print-reply --dest=org.mpris.MediaPlayer2.spotify /org/mpris/MediaPlayer2 org.mpris.MediaPlayer2.Player.Stop" XF86AudioStop
"dbus-send --print-reply --dest=org.mpris.MediaPlayer2.spotify /org/mpris/MediaPlayer2 org.mpris.MediaPlayer2.Player.Previous" XF86AudioPrev
"dbus-send --print-reply --dest=org.mpris.MediaPlayer2.spotify /org/mpris/MediaPlayer2 org.mpris.MediaPlayer2.Player.Next" XF86AudioNext
</code></pre>
<p>For those who are interested, there’s a more complete exploration of the available commands <a href="http://www.mabishu.com/blog/2010/11/15/playing-with-d-bus-interface-of-spotify-for-linux/" title="Playing with D-Bus interface of Spotify for Linux">at Mabishu</a>, and a tool which makes use of the API for track notifications <a href="http://muffinresearch.co.uk/archives/2011/03/23/linux-spotify-track-notifier-with-added-d-bus-love/" title="Linux: Spotify Track Notifier with added D-Bus love">at Muffin Research Labs</a>.</p>
<p><strong>UPDATE:</strong></p>
<p><a href="http://www.noswap.com/projects/spotify-gnome/" title="Spotify Gnome Integration: NoSwap.com">John Reese</a> has implemented a program to automatically provide the mappings needed to control Spotify over DBUS. You can get it from <a href="https://github.com/jreese/spotify-gnome" title="spotify-gnome: GitHub">GitHub</a>.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Lock-Free IPC]]></title>
            <link>https://kothar.net/blog/lock-free-ipc</link>
            <pubDate>Wed, 16 Mar 2011 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/lock-free-ipc</guid>
            <description>
              <![CDATA[<p>I was doing some reading a while back into <a href="http://drdobbs.com/go-parallel/article/219500200" title="Practical Lock-Free Buffers">lock-free algorithms</a> for multi-threaded communication using buffers. The general idea is to use a circular buffer in shared memory to allow one process (or thread) to write data into a buffer so that another process can read it. The algorithm relies on atomic <a href="http://en.wikipedia.org/wiki/Compare-and-swap" title="CAS (Compare and Swap)">compare-and-swap</a> instructions, which are part of the x86 instruction set and also present in several other architectures which have features designed to assist with concurrency.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>I was doing some reading a while back into <a href="http://drdobbs.com/go-parallel/article/219500200" title="Practical Lock-Free Buffers">lock-free algorithms</a> for multi-threaded communication using buffers. The general idea is to use a circular buffer in shared memory to allow one process (or thread) to write data into a buffer so that another process can read it. The algorithm relies on atomic <a href="http://en.wikipedia.org/wiki/Compare-and-swap" title="CAS (Compare and Swap)">compare-and-swap</a> instructions, which are part of the x86 instruction set and also present in several other architectures which have features designed to assist with concurrency.</p>
<p>It turns out that one of my colleagues has already <a href="https://www.rapitasystems.com/blog/how-set-safe-portable-interprocess-communication-without-interrupt-locks" title="How to set up safe, portable interprocess communication without interrupt locks">written about this</a> on the Rapita blog - it’s a useful approach, and used in many systems.</p>
<h2>IPC</h2>
<p>There are some issues with using shared memory for IPC: shared memory is generally considered to be ‘dangerous’ in some way, since it punctures the memory isolation principles of most operating system kernels. It is therefore carefully controlled by the OS.</p>
<p><a href="http://fscked.org/writings/SHM/shm.html" title="Shared Memory Under Linux">Linux supports</a> shared memory by creating shared memory ‘files’ (segments) which are then mapped into the address space of one or more processes. These segments must be cleaned up after use however, otherwise memory leaks in the Kernel occur.</p>
<p>The design principles of a <a href="http://en.wikipedia.org/wiki/Microkernel#Inter-process_communication" title="Microkernel">microkernel</a> often demand complete memory isolation - the first generation microkernels had very poor performance because of this dilemma: on the one hand they demanded isolation, but this prevented high-performance communication between processes. On the other hand by moving system services out of the kernel and into independent processes (Servers) they vastly increased the amount of IPC needed for the basic operation of the system.</p>
<p>Later designs have allowed higher performance by internally using something akin to shared memory to implement their message-passing IPC. The <a href="http://en.wikipedia.org/wiki/Mach_(kernel)" title="Mach Kernel">Mach kernel</a> uses a mechanism where the memory containing the message is mapped into the receiving process’ address space. By flagging the memory pages as read-only, the operating system can step in and <a href="http://en.wikipedia.org/wiki/Copy-on-write" title="Copy-on-write">make a new copy</a> of the memory if the receiving process tries to modify it.</p>
<p>The documentation for the QNX kernel describes why shared memory alone <a href="http://www.qnx.com/developers/docs/6.3.2/neutrino/sys_arch/kernel.html#id31" title="The QNX Neutrino Microkernel">isn’t enough for all IPC mechanisms</a>, since in the end some form of synchronisation will be required, and the process must fall back on the OS to provide this.</p>
<h2>Unidirectional buffer</h2>
<p>It strikes me that there ought to be a middle ground, similar to the approach used by the Mach microkernel, but that doesn’t require so many system calls to handle passing messages. My current idea is that to create a producer-consumer buffer between two processes, a process requests the kernel to create a new channel (or Port in microkernel terminology) to the receiving process. This will allocate a number of memory pages in both processes which are read/write for the producer and read-only for the consumer. Now we can implement the circular buffer mechanisms as described in the Rapita article, except that we still need at least one variable to be writeable by the consumer, so that it can signal to the producer the current position of the read pointer. This could be accomplished by mapping a single shared memory page which is writable only by the consumer, to store the pointer.</p>
<p>For a single variable this seems extravagant - a page is usually 4kb in size in current operating systems, but could be 4mb on some systems.</p>
<p>Other concerns include how to handle buffers between processes on separate cores (the ideal use case for lock-free buffers) or separate systems in a network. I think it <em>ought</em> to work in the first case, but I suspect extra work needs to be done to flush memory pages from cache - the CAS instruction ought to at least force its own cache line to be flushed, but the rest of a message might not be.</p>
<p>For networked buffers, the best solution might be a ‘dummy’ or stub process to be created which simply streams the content of the buffer over the network. This would be transparent to the sending process, and a corresponding stub on the receiving system should provide the same effect to the receiver. I have concerns about latency and so on, but if the buffer can be kept sufficiently empty the throughput should be high.</p>
<p>Finally, the receiver would have to be careful to make its own copy of any data before modifying it - the read-only pages of the buffer would otherwise cause a segmentation fault. This might be worked around in the same way as the Mach message passing IPC using copy-on-write mechanisms, but would then incur the extra overhead of system calls to perform the memory mapping as each message is received.</p>
<h2>Conclusion</h2>
<p>I’ll almost certainly be experimenting with this approach in the future, and will be interested to see the relative performance of the different approaches.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Enabling IMAP SSL and TLS encryption for mail-notifier under Ubuntu]]></title>
            <link>https://kothar.net/blog/tls-mail-notifier</link>
            <pubDate>Mon, 14 Dec 2009 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/tls-mail-notifier</guid>
            <description>
              <![CDATA[<p>After struggling all morning to find or compile a mail notification applet that supports TLS encryption under Ubuntu, I
finally read all the way to the end of
the <a href="https://bugs.launchpad.net/ubuntu/+source/mail-notification/+bug/44335?comments=all">Launchpad bug for mail-notifier</a> (
all 119 comments) and found <a href="https://launchpad.net/~jurenka">David Jurenka</a>’s
time-saving <a href="https://launchpad.net/~mail-notification-ssl/+archive/ppa">PPA with SSL support enabled</a>.</p>
<p>Thanks David!</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>After struggling all morning to find or compile a mail notification applet that supports TLS encryption under Ubuntu, I
finally read all the way to the end of
the <a href="https://bugs.launchpad.net/ubuntu/+source/mail-notification/+bug/44335?comments=all">Launchpad bug for mail-notifier</a> (
all 119 comments) and found <a href="https://launchpad.net/~jurenka">David Jurenka</a>’s
time-saving <a href="https://launchpad.net/~mail-notification-ssl/+archive/ppa">PPA with SSL support enabled</a>.</p>
<p>Thanks David!</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Shorter Bash Prompts]]></title>
            <link>https://kothar.net/blog/shorter-bash-prompts</link>
            <pubDate>Fri, 04 Sep 2009 22:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/shorter-bash-prompts</guid>
            <description>
              <![CDATA[<p>I use Java a lot, and the common practice of naming java packages with long URI-derived namespaces often leads to my bash prompt becoming extremely long. Here’s a script to display an abbreviated path on the bash prompt.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>I use Java a lot, and the common practice of naming java packages with long URI-derived namespaces often leads to my bash prompt becoming extremely long. Here’s a script to display an abbreviated path on the bash prompt.</p>
<p>The prompt definitions and script below will display a shortened string derived from the path by taking the first letter from each path segment, and the last segment in its entirity.</p>
<p>For example, the following terminal prompt inside a hypothetical Eclipse plugin I might be developing:</p>
<pre><code>michael@kothar ~/dev/workspace/net.kothar.shiny/src/net/kothar/shiny/actions$
</code></pre>
<p>would be abbreviated as:</p>
<pre><code>michael@kothar ~/d/w/n/s/n/k/s/actions$
</code></pre>
<p>To most that probably doesn’t mean much, but I think most developers have a good idea what’s in their directory structures, and can probably work out pretty quickly where they are. There’s always a call to the trusty pwd command if you are feeling lost. In some projects I have at work, the package names are even longer, so the reduced amount of text to parse is quite welcome.</p>
<p>Here’s what you need:</p>
<p>New prompt definition (with colours):</p>
<pre><code>PS1="${debian_chroot:+($debian_chroot)}\[\033[00;32m\]\u@\h\[\033[00m\]:\[\033[00;34m\]\$(shortpath)\[\033[00m\]\$ "
</code></pre>
<p>Or more simply (without colours):</p>
<pre><code>PS1="\$(shortpath)\$ "
</code></pre>
<p>shortpath:</p>
<pre><code>#!/usr/bin/perl  

use Cwd;  
my $path = &#x26;Cwd::cwd();  

# Replace the 'home' segment with a tilde  
$path =~ s/^$ENV{'HOME'}/~/;  

my @segments = split(/\//, $path);  

my $shortpath = "";  
while (my $segment = shift @segments) {  
	if (@segments) {  
		$shortpath .= substr ($segment, 0, 1) . "/";  
	}  
	else {  
		$shortpath .= "$segment";  
	}  
}  
print $shortpath;
</code></pre>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Javada 0.1.1]]></title>
            <link>https://kothar.net/blog/javada-0.1.1</link>
            <pubDate>Mon, 02 Feb 2009 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/javada-0.1.1</guid>
            <description>
              <![CDATA[<p>I’ve released a new version of Javada today - just a maintenance fix, after an email request for a pre-compiled version. It’s available for download from:</p>
<p><a href="https://bitbucket.org/mikehouston/javada/downloads">https://bitbucket.org/mikehouston/javada/downloads</a></p>
<p>Also note, the licence is now LGPL.</p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>I’ve released a new version of Javada today - just a maintenance fix, after an email request for a pre-compiled version. It’s available for download from:</p>
<p><a href="https://bitbucket.org/mikehouston/javada/downloads">https://bitbucket.org/mikehouston/javada/downloads</a></p>
<p>Also note, the licence is now LGPL.</p>
]]>
            </content:encoded>
          </item>
    
          <item>
            <title><![CDATA[Brief introduction to PolyORB]]></title>
            <link>https://kothar.net/blog/brief-introduction-to-polyorb</link>
            <pubDate>Mon, 10 Nov 2008 00:00:00 GMT</pubDate>
            <guid>https://kothar.net/blog/brief-introduction-to-polyorb</guid>
            <description>
              <![CDATA[<p>I’ve been experimenting with PolyORB at work, and the documentation, while fine for configuring and compiling your application, is sorely lacking on any help with implementing your CORBA servants.</p>
<p>Here I’ll attempt to provide some brief hints on how to do things that don’t seem to be described in the documentation. The <a href="https://web.archive.org/web/20160603114356/http://lists.adacore.com/pipermail/polyorb-users/" title="The PolyORB-users Archives">PolyORB mailing list</a> is a good place to find more information.</p>
<p><em>[Nov 2021]: The mailing list seems to have been defunct since around 2016</em></p>
]]>
            </description>
            <content:encoded>
              <![CDATA[<p>I’ve been experimenting with PolyORB at work, and the documentation, while fine for configuring and compiling your application, is sorely lacking on any help with implementing your CORBA servants.</p>
<p>Here I’ll attempt to provide some brief hints on how to do things that don’t seem to be described in the documentation. The <a href="https://web.archive.org/web/20160603114356/http://lists.adacore.com/pipermail/polyorb-users/" title="The PolyORB-users Archives">PolyORB mailing list</a> is a good place to find more information.</p>
<p><em>[Nov 2021]: The mailing list seems to have been defunct since around 2016</em></p>
<h2>Well Known Services</h2>
<p>I wanted to be able to use a simple corbaloc with an object ID which doesn’t change, along the lines of:</p>
<pre><code>'corbaloc:iiop:1.2@192.168.1.20:2809/MyServant'
</code></pre>
<p>This <a href="https://web.archive.org/web/20160603114356/http://lists.adacore.com/pipermail/polyorb-users/2008-July/000999.html" title="Activate_Object_With_Id">doesn’t seem</a> <a href="https://web.archive.org/web/20160603114356/http://lists.adacore.com/pipermail/polyorb-users/2008-June/000993.html" title="Interoperability of PolyOrb with other CORBA implementations">to be possible</a> as of yet, but <a href="https://web.archive.org/web/20160603114356/http://lists.adacore.com/pipermail/polyorb-users/2008-July/001004.html" title="Initiate Well Known Service">this message</a> suggests that the easiest thing to do is to use a “Well Known Service” instead.</p>
<p>You can initialise a new well-known service using the following code:</p>
<pre><code>declare  
   Ref : CORBA.Object.Ref;  
   Obj : constant CORBA.Impl.Object_Ptr := new Service.Impl.Object;  

begin  
   PolyORB.CORBA_P.Server_Tools.Initiate_Well_Known_Service  
      (PortableServer.Servant (Obj), "MyServant", Ref);  
   -- Now do something with Ref;  
end;
</code></pre>
<p>Which will produce a corbaloc which looks like this:</p>
<pre><code>'corbaloc:iiop:1.2@192.168.1.20:2809/MyServant/000000024fF0000000080000000'
</code></pre>
<p>It’s not quite as concise, but it is always the same.</p>
<h2>Binding Interfaces</h2>
<p>Telling PolyORB which interfaces to bind to is still something of a mystery to me. If you don’t specify a default address, the ORB will bind to every interface. The problem comes when the corbaloc now uses the ‘default’ address for your hostname (specified in the hosts file), which in my case is <code>127.0.1.1</code>. This isn’t much help when you want to connect to a remote computer.</p>
<p>To specify a default address manually, you need to create a polyorb.conf file. Mine looks like this:</p>
<pre><code>###############################################################  
# IIOP Global Settings  

[iiop]  

# Preference level for IIOP  
polyorb.binding_data.iiop.preference=1  

# IIOP's default address  
polyorb.protocols.iiop.default_addr=192.168.1.20  

# IIOP's default port  
polyorb.protocols.iiop.default_port=2809  

# IIOP's alternate addresses  
polyorb.protocols.iiop.alternate_listen_addresses=127.0.0.1:2809  

# Default GIOP/IIOP Version  
polyorb.protocols.iiop.giop.default_version.major=1  
polyorb.protocols.iiop.giop.default_version.minor=2
</code></pre>
<p>This file needs to be in the current directory - not in the path, not with the executable, but in the current working directory.</p>
<p>I’m still working on a way to deal with dynamically assigned IP addresses. Ideally, the corbaloc would include the hostname rather than an IP address, giving us something like this:</p>
<pre><code>'corbaloc:iiop:1.2@kothar:2809/MyServant/000000024fF0000000080000000'
</code></pre>
<p>but that has its own problems (the host name should be fully qualified if you expect it to be found on the internet for example).</p>
<p>I’ve no idea how to get the utilities to generate the correct references for me, so I’d have to manually edit them to use the host name. The <code>Object_To_Corbaloc</code> function simply returns the corbaloc for the “best profile” - whatever that is.</p>
<p>Looking at the implementation, this is based entirely on transport preference levels, so i can’t see an easy way to prefer one address over another except manually specifying the default.</p>
<p>Another huge problem is that even though the ORB binds to all interfaces, any IORs it generates use the default IP address! So if I leave the default as <code>127.0.1.1</code>, but connect to <code>192.168.1.20</code>, any object references passed to the client from the server will use <code>127.0.0.1</code>. Not very useful.</p>
<p>You can read more about the problem in <a href="https://web.archive.org/web/20160603114356/http://lists.adacore.com/pipermail/polyorb-users/2005-January/000084.html">this post</a> on the mailing list.</p>
<h2>Using Sequences</h2>
<p>Sequences were a bit confusing to me at first, since I’m more familiar with the Java bindings which map everything to standard generic List objects. This is mostly just to provide an example of using them.</p>
<p>In PolyORB, each sequence is mapped to a new generic package which, for IDL-defined interfaces at least, contains ‘Forward’ objects. Each reference must be converted to its equivalent Forward before it can be inserted into the sequence. This is done using the ‘Convert’ package defined with each interface.</p>
<p>That’s probably not very clear, but below is an example.</p>
<pre><code>with jobs;  
with jobs.Helper;  
with jobs.Job.Helper;  
with jobs.Job.Impl;  
with PolyORB.Sequences.Unbounded;  

function newJob  
     (Self : not null access Object;  
      config : jobs.JobConfig)  
     return jobs.Job.Ref  
   is  
      Root_POA : PortableServer.POA.Local_Ref :=  PolyORB.CORBA_P.Server_Tools.Get_Root_POA;  

      Obj : constant jobs.Job.Impl.Object_Ptr := new jobs.Job.Impl.Object;  
      Ref : CORBA.Object.Ref;  

      Result : jobs.Job.Ref;  

      use jobs.Job.Convert_Forward;  
   begin  
      -- Register new servant with Root POA  
      Ref := PortableServer.POA.Servant_To_Reference  
        (Root_POA, PortableServer.Servant (Obj));  

      -- Downcast to Job reference  
      Result := jobs.Job.Helper.Unchecked_to_ref(ref);  

      -- Append to list of current jobs  
      Append(Self.Job_List, To_Forward(Result));  

      return Result;  
   end newJob;
</code></pre>
]]>
            </content:encoded>
          </item>
    
        </channel>
      </rss>