Prism: Ruby 3.3's new error-tolerant parser
Prism is a new library shipping as a default gem in Ruby 3.3.0 that provides access to the Prism parser, a new parser for the Ruby programming language. Prism is designed to be error tolerant, portable, maintainable, fast, and efficient.
Advent of Prism: Part 24 - Error tolerance
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about error tolerance.
Advent of Prism: Part 23 - Pattern matching (part 2)
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about pattern matching.
Advent of Prism: Part 22 - Pattern matching (part 1)
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about pattern matching.
Advent of Prism: Part 21 - Throws and jumps
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about throws and jumps.
Advent of Prism: Part 20 - Alias and undef
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about the alias and undef keywords.
Advent of Prism: Part 19 - Blocks
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about blocks and lambdas.
Advent of Prism: Part 18 - Parameters
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about parameters.
Advent of Prism: Part 17 - Scopes
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about nodes that introduce a new scope.
Advent of Prism: Part 16 - Control-flow calls
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about control-flow calls.
Advent of Prism: Part 15 - Call arguments
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about call arguments.
Advent of Prism: Part 14 - Calls (part 2)
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about call nodes.
Advent of Prism: Part 13 - Calls (part 1)
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about call nodes.
Advent of Prism: Part 12 - Program structure
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about program structure.
Advent of Prism: Part 11 - Defined
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about the defined? keyword.
Advent of Prism: Part 10 - Regular expressions
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about regular expressions.
Advent of Prism: Part 9 - Strings
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about strings.
Advent of Prism: Part 8 - Target writes
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about target writes.
Advent of Prism: Part 7 - Control-flow
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about control-flow.
Advent of Prism: Part 6 - Control-flow writes
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about control-flow writes.
Advent of Prism: Part 5 - Operator writes
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about operator writes.
Advent of Prism: Part 4 - Writes
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about writes.
Advent of Prism: Part 3 - Reads
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is about reads.
Advent of Prism: Part 2 - Data structures
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is data structures.
Advent of Prism: Part 1 - Literals
This blog series is about how the prism Ruby parser works. If you’re new to the series, I recommend starting from the beginning. This post is literals, and their variations.
Advent of Prism: Part 0 - Introduction
Two and a half years ago, I began working on what is today known as the prism Ruby parser. This is a new Ruby parser that is now being integrated into runtimes and tooling alike. This includes CRuby, JRuby, TruffleRuby, Natalie, Syntax Tree, RuboCop, and many others.
ActiveRecord::UnionRelation
I wrote a gem back in 2020 called active_record-union_relation, and I’m finally writing a blog post describing it. This is a very small gem that provides the ability to define Active Record relations from UNION queries. The resulting records that are returned when the queries are executed are polymorphic (i.e. they can be instances of different classes). I’ll show how this works and why you might want to use it.
Ruby operators
Ruby’s grammar has a ton of operators. Overtimes, they can mean more than one thing, depending on their context. This blog post enumerates each operator and its meaning.
Rewriting the Ruby parser
At Shopify, we have spent the last year writing a new Ruby parser, which we’ve called YARP (Yet Another Ruby Parser). As of the date of this post, YARP can parse a semantically equivalent syntax tree to Ruby 3.3 on every Ruby file in Shopify’s main codebase, GitHub’s main codebase, CRuby, and the 100 most popular gems downloaded from rubygems.org. We recently got approval to merge this work into CRuby, and are very excited to share our work with the community. This post will take you through the motivations behind this work, the way it was developed, and the path forward.
2023 Resolutions
In the back of my head, I’ve always had a bunch of open-source projects that I wanted to build if I had the time. This has led to me sporadically working on a wide variety of projects without any real clear direction. This coming year I have decided to try to focus on specific goals in the hopes that it will help me stay more on track and actually complete a project for once. In an effort to be more transparent and maybe incentive others to help me reach them, I’m sharing them here. Without further ado, here are my 2023 open-source resolutions, in their order of priority.
Advent of YARV: Part 24 - Wrap up
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This is the last post in the series.
Advent of YARV: Part 23 - Primitive
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about the Primitive module and the invokebuiltin instruction.
Advent of YARV: Part 22 - Pattern matching
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about pattern matching.
Advent of YARV: Part 21 - Once
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about the once instruction.
Advent of YARV: Part 20 - Catch tables
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about catch tables.
Advent of YARV: Part 19 - Defined
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about the defined? keyword.
Advent of YARV: Part 18 - Super methods
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about super methods.
Advent of YARV: Part 17 - Method parameters
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about method parameters.
Advent of YARV: Part 16 - Defining methods
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about defining methods.
Advent of YARV: Part 15 - Defining classes and modules
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about defining classes and modules.
Advent of YARV: Part 14 - Branching
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about branching.
Advent of YARV: Part 13 - Constants
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about constants.
Advent of YARV: Part 12 - Global variables
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about global variables.
Advent of YARV: Part 11 - Class and instance variables
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about how YARV handles class and instance variables.
Advent of YARV: Part 10 - Local variables (3)
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is the last of three posts about local variables.
Advent of YARV: Part 9 - Local variables (2)
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is the second of three posts about local variables.
Advent of YARV: Part 8 - Local variables (1)
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is the first of three posts about local variables.
Advent of YARV: Part 7 - Calling methods (2)
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is the second of two posts about calling methods.
Advent of YARV: Part 6 - Calling methods (1)
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is the first of two posts about calling methods.
Advent of YARV: Part 5 - Changing object types on the stack
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about changing object types on the stack.
Advent of YARV: Part 4 - Creating objects from the stack
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about creating objects from the stack.
Advent of YARV: Part 3 - Frames and events
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about frames and events.
Advent of YARV: Part 2 - Manipulating the stack
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about manipulating the virtual machine stack.
Advent of YARV: Part 1 - Pushing onto the stack
This blog series is about how the CRuby virtual machine works. If you’re new to the series, I recommend starting from the beginning. This post is about the virtual machine stack, and how to push values onto it.
Advent of YARV: Part 0 - Introduction
Since I started working on the YJIT team at Shopify, I’ve been learning more and more about the CRuby virtual machine known as YARV. A lot of the details of how YARV works are not well documented or the documentation is difficult to find. As such, I decided to write a series of blog posts about how YARV works internally as a Christmas present to both the Ruby community and myself. I hope that this series will help others understand how YARV works and provide a better understanding of CRuby internals. This is the blog series I wish I had had access to when I first started working on CRuby.
AArch64 Bitmask Immediates
This post illustrates a small but fascinating piece of the AArch64 architecture called bitmask immediates. We’ll briefly cover what AArch64 is, how it is different from other architectures, what a bitmask immediate is, and how all of this can be encoded in Rust.
Syntax Tree and lambda-local variables
I just released version 2.6.0 of Syntax Tree. Along with a couple of other changes, this includes support for lambda-local variable declarations. This was a bit of a journey, so I thought I’d write up how I discovered this syntax, how I added support for it to Syntax Tree, and go ahead and plug Syntax Tree one more time as something that should be merged into Ruby core.
ruby-syntax-tree.github.io
Over the weekend I cobbled together ruby-syntax-tree.github.io, and I thought I’d share a quick post about what it is, how it works, and what I learned while I built it.
Ruby Association Final Report
In accordance with the Ruby Association’s timeline, this is the final report on the Ruby formatter project.
Formatting Ruby: Part 1 - How ripper works
This post is part of a series about how ruby-syntax-tree/syntax_tree works under the hood. It’s part of an ongoing effort to spread the word about this project, document how it works, and explain some of the internals for anyone interested. This post specifically focuses on ripper, the Ruby standard library used to parse Ruby files. For an overview of this series, see the introduction post.
Formatting Ruby: Part 0 - Introduction
Last October, the Ruby Association selected its 2021 grant recipients for the various projects around the Ruby ecosystem that they would support. Among them was my proposal to create a standard library Ruby formatter. Below is the description of the project, as per the submission:
Solving Wordle in Ruby
You may have seen the word game wordle going around the various social media. It’s a simple game that works like this:
Ruby Association Intermediate Report
In accordance with the Ruby Association’s timeline, this is an intermediate report on the Ruby formatter project.
Ripper CHANGELOG 3.1.0
The Ripper module ships with the Ruby standard library and gets updated (implicitly or explicitly) every time the Ruby parser changes. Unfortunately, Ripper itself never changes version (it’s been stuck at 0.1.0 since it was first shipped with Ruby in 2004). As such, there isn’t really a dedicated CHANGELOG, and it’s somewhat difficult to determine what changed inside the Ripper module without digging into the source.
Prettier Ruby 2.0.0
I just released the 2.0.0 version of the prettier plugin for Ruby. In this post I’m going to talk about what this project is, how it works, what the 2.0.0 release means, and where this project is going.
Ruby type conversion
Let’s talk about type conversion in Ruby.
Linting Ruby
The last couple of weeks I’ve been thinking about the process of linting Ruby code. For the most part within the Ruby community we’ve pretty much standardized on rubocop, and for good reason - it’s an impressive project with a massive breadth in terms of how far it is willing to go to guide you toward better code. Other tools within the community exist as well, including my personal favorite reek.
Prettier's encoding bug
About a month ago, a curious bug was reported on the prettier plugin for Ruby in this issue: prettier/plugin-ruby#596. It stated that the parser was failing with an error saying that there was an invalid byte sequence for US-ASCII encoding. What made things even more confusing was that it only occured when it was being run through the JavaScript side, not just through the Ruby side.
Polyfill as needed
Because browsers implement their own versions of JavaScript, they don’t all support the same functions. For example, Array#includes is supported in all of the major browsers except Internet Explorer. So, if you include that function in your application (or one of your dependencies does) your application will break in Internet Explorer only.
react-state-mutations
There are two types of objects that you can pass to the first argument to setState within React components. The first is an object, which will update the state to be equal to that value, as in:
Weird Ruby
Recently, I wrote a plugin for prettier for the Ruby programming language. Over the course of that process, I discovered a lot of eccentricities of Ruby (by needing to account for each node type of, as well as each variant in structure of, the AST). I found some fun things, and so in the spirit of Gary Bernhardt’s “wat” talk, I’m going to share them with you here.
Dynamic ActiveRecord columns
When dealing with any kind of data transmission, the problem almost always boils down to the amount of data being sent. Engineers can optimize the speed of the wire and the compression of the data, but at the end of the day sending fewer things is almost always going to yield the biggest gains.
Delegating ActiveRecord scopes
In Ruby on Rails, access to the database is by default controlled through the ActiveRecord ORM. ActiveRecord operates on Relation objects that contain the configuration for an SQL query that will be executed at a later time. Relation objects have the ability to copy their internal configuration over to a new Relation object with new options in order to further customize those queries.
Skipping preflight checks
Preflight checks are part of the CORS system that browsers fire before cross-origin requests in order to determine if the request is allowed. These requests can add up over time and cause a discernable lag in your web application.
Faster Rails tests
Feedback loop speed in one of the biggest contributing factors to overall development time. The faster you get results, the faster you can move on to other things. A fast enough test suite is therefore critical to teams’ success, and is worth investing some time at the beginning to save in the long run.
Infrastructure as code with humidifier-reservoir
A little more than a year ago we made the first commit to the gem that eventually became humidifier. It’s evolved quite a bit in the last year, including integrating AWS’ resource specification which had the side-effect of greatly stabilizing the API. Here at Localytics, we’ve been using humidifier in myriad ways, including managing our AWS infrastructure, launching and configuring new servers, and aiding in refactoring our network ACLs for better security (to name a few).
Connecting to Snowflake with Ruby on Rails
At Localytics, one of the tools we use for data processing is the Snowflake data warehouse. We connect to Snowflake in a couple different ways, but our main data retrieval application is a Ruby on Rails API. To accomplish this we use a combination of unixODBC (an open-source implementation of the ODBC standard), Snowflake’s ODBC driver, and our own ODBC ActiveRecord adapter for Ruby on Rails. This sequence of tools allows us to take full advantage of ActiveRecord’s query generation and general ease-of-use while still enjoying all the benefits of a fully cloud-enabled data warehouse such as Snowflake.
Cogito - Adding "I think" to IAM
Today we are open-sourcing Cogito, an abstraction of AWS IAM syntax. IAM (Identity and Access Management) is AWS’s module that dictates the ability of various users and resources to mutate other resources within the AWS ecosystem. Permissions are described through a JSON-structured policy document that lives within the AWS console.
ODBC and writing your own ActiveRecord adapter
Today we are open-sourcing our ODBC adapter for ActiveRecord, which allows Ruby on Rails applications to communicate with ODBC-compliant databases. The impetus for this work was an effort to update one of our APIs to run with the latest Rails and ruby. Along the way we released Rails 3.2.x, 4.2.x, and 5.0.x versions of the adapter, along with deploying incremental upgrades to our API as we went. Below is the story of how we made it happen.
Best Practices and Common Mistakes with Travis CI
At Localytics, we’ve worked hard to ensure that all of our services are built and deployed using our own internal best practices. One of those core best practices is that the service in question is using continuous integration (CI). CI greatly reduces the amount of time an engineer needs to spend finding the origin of a bug by running tests early and often. By integrating each change into a central repository where the tests can be run, we can precipitate faster cycle times and a more productive work environment.
Exploring CLI best practices
Like at many software companies, we at Localytics build command-line interfaces (CLIs) that manage our internal infrastructure and processes. These tools cover a broad range of applications, including integrating with our background jobs server, creating and maintaining AWS resources, as well as handling deployment and continuous delivery. We’ve written all of our CLIs with ruby, using thor to handle parsing and dispatching the commands.
Humidifier - CloudFormation made easier
Today we are open-sourcing Humidifier - one of the tools that we use internally to manage infrastructure on Amazon Web Services (AWS) at Localytics. This Ruby gem allows you to programmatically generate domain objects that represent the AWS CloudFormation (CFN) resources you want, handling the details of JSON manipulation under the covers. We’ve found that Humidifier not only increases development speed, but also results in easy-to-understand, maintainable code that allows you to focus on building resources instead of programming in JSON.
Serverless Slackbots powered by AWS
Today Localytics is open sourcing two tools that help you quickly scaffold custom Slack commands using AWS Lambda, AWS API Gateway, and the Serverless framework. These tools grew out of our continued investment in making ChatOps the primary way Localytics engineers manage our infrastructure. With Serverless and a small bit of scaffolding, we’re now able to create custom Lambda functions that handle slash commands from Slack in about 10 minutes.