I'm a little worried on behalf of the "Python Language Tooling Team" at Meta, because uv has been so popular, and I wouldn't be surprised if ty wins out in this space.
So watch out, or this will become like Atom or Flow, an internal competitor of a technology that is surpassed by the more popular external open source version, leaving the directors/vps muttering to themselves "It's too bad that this team exists at all. Could we get rid of them and just switch to the open source stuff?"
Perhaps just something for the manager (Aaron Pollack?) to keep an eye on....
Hey Kevin, we overlapped for a bit during your time at FB when I was working on Flow. Nice to hear from you!
I’m working on Pyrefly now, but worked on Flow for many years before. For what it’s worth, we are taking a different approach compared to Flow and have explicitly prioritized open source and community building, something I know we both care a lot about.
Of course, nothing is guaranteed and we’ve seen plenty of volatility in bigco investments to infra lately, but I do believe we’re starting this journey on the right foot.
Meta seems to place a pretty high premium on controlling its open source projects, especially dev tooling. I guess dating back at least to the git maintainers telling them they were doing things wrong with their monorepo and refusing to upstream scale fixes, which precipitated their migration to mercurial (who were more than happy to take the contributions).
Given the change velocity of internal tooling you can understand why owning your own project makes sense here.
Of course tagged string literals can be compiled down to a form where no string parsing occurs. This is a really old technique; I implemented it, like many others, in PHP in late 1990s.
Vut inside an IDE or an LSP, tagged literal strings need quite a bit more special support than JSX, AFAICT.
I believe the argument was that jsx doesn't support that either, which should be correct as far as I know.
They're all features in the jsx ecosystem that are provided by other libs, which are often preconfigured with projects utilizing jsx.
I did miss a lot of the tooling I've gotten used to the last time I made a weekend project utilizing lit webcomponents for sure, but I believe the people advocating for it usually rejoice in having less tooling in their development pipeline - it's just a fundamental disconnect between people that have learned and gotten used to current frontend development and people that don't really want to bother with it, looking for simpler solutions with less moving parts.
Hi folks, I work on the Pyrefly team at Meta. Our FAQ covers a good number of the questions raised here: https://pyrefly.org/en/docs/pyrefly-faq/. I can also try to answer some of your questions. Thanks for taking a look!
There's nothing magical to type-checking Python. You can write this in any programming language. TypeScript is actually a pretty nice language for writing static analysis tools.
Yes. If you want runtime validation of data you’re taking in people recommended pydantic. If you’re looking for runtime validation within your own code I’ve seen people use beartype, though to be honest I don’t personally understand the value added from it
On one hand, I feel like I've been in a coma since covid because I've just been coasting along with Marshmallow and jsonschema, but on the other hand it's like a lot of the major advances have been in the past couple years. Apparently pydantic got a big version update in 2023? And now all these competing static type checkers?
"Today we’re releasing Pyrefly as an alpha. At the same time, we’re busy burning down the long-tail of bugs and features aiming to remove the alpha label this Summer. "
The Rust code written here is so easy to follow but all these new Python tooling being written in Rust worries me, it adds yes another vector to the N-language problem.
For the Python ecosystem, it's natural to use Python where Python can cope, and a high-performance language where it cannot. There are two such languages in wide use around Python: Rust, and, inevitably, C. So N = 3.
(C, to my mind, should be eventually phased out from application programming altogether, so N would be 2, but it's a loooooong process; Python may become a legacy language before it converges.)
Why is "written in Rust" a feature to be mentioned? Who cares? So my type checker has memory protection and is compiled. I'm not running my type checker on an embedded system or in a mission critical service. It seems kind of like "written in Erlang". I'd prefer to have non-performance critical code for Python written in Python. That way the broader community can support and extend it.
Have you used Rust before? As a user, the speed and safety is nice. But as a developer, Rust projects are easier to hack on and contribute to.
That's kind of the whole appeal of Astral. I know Python better than Rust, but it's a lot easier for me to hack on Rust projects. The whole appeal of Astral is that they want to bring Rust-quality tooling to Python.
Static typing is a big one, but I've been so steeped in Python that I don't appreciate it as much as maybe I should.
The big thing for me is that most Rust projects are statically(ish) compiled (like go) and only need a `cargo build`. No venvs or pip commands or node/npm/nvm or make, etc.
Also, the Zen of Python is supposed to be one obvious way to do something but ddg "python dependency manager" and have a good laugh. That actually becomes triple lol for when uv steps in and says "uh, your pip requirements.txt is unsatisifyable, what are you even?!"
And let me just say what an absolutely stellar job the rustc authors have done with the error messages. Just out of this world thoughtful. I have not yet gotten on board with their compile time vs safety and insight tradeoff, but I can see why they chose those two axes
Type inference is handy, but as an observation source code is 90% written for humans - I believe that CPython knows what types they are, that's how TypeError even exists, but seeing "def doit(erthang):" doesn't help me in the slightest. Often PyCharm can figure it out, because bless their hearts, but then developers whine about "wwwaaa, pycharm too slo grug use ed" and then one gets AttributeError in production because who could have foreseen
Pyright being so fast has gotten my coworkers (as well as myself) who dare to code using tooling not made by JetBrains an easy to use instant-feedback type checker that can be used in CI. So I think it's a huge win.
This response highlights both parts what I was saying: it's not just strings, and "I'm sure there's some extra things that the standard library wants you to duct tape together, good luck"
It is exactly two strings that are required. The docs are there for flags etc, use them. Types were not entered directly into stdlib source for historical reasons.
If you would enjoy further support, install stubs from typeshed.
$ python3.12 -c '
import re
pat = re.compile("who knew")
ma = re.match(pat, "who knew types matter")
print(ma)
'
<re.Match object; span=(0, 8), match='who knew'>
> The docs are there for flags etc, use them.
I guess it's good we're all using LLMs nowadays, since in general computers no read so good, that's why we write in specialized languages for their benefit. That would include this fancy new thing I've heard about where one writes down what the input and output domains are for functions
> Types were not entered directly into stdlib source for historical reasons.
Historical reasons defeats the purpose of having git tags, to say nothing of them having several concurrent branches named after the various release trains. I mean, historically print was a keyword, but you sure don't see them from __future__ import print_statement all over the 3.12 tree now do you? It's because they DGAF preferring there to be seemingly unlimited aftermarket tooling to try and drag python3000 into the 21st century
> If you would enjoy further support, install stubs from typeshed.
While trying to dig up whatever a sane person would use to "install stubs" -- because it for damn sure isn't $(pip install typeshed) -- I learned that you were even more incorrect - it accepts bytes, too https://github.com/python/typeshed/blob/9430260770b627c04313...
Anyway, they also go out of their way to say "don't install typeshed" and I guess that's why they don't have any git tags because releases are for children
Right, the first arg can also be a compiled pattern|bytes. Not an issue in the real world however. The problem is when it can't handle what you give it, not when it can handle what you give it and additional things.
Normally you'd use `pattern.match(string)` in that case.
They said specifically more than once that they're not going to change (almost) every line in the repo for any reason, and not only to types but other syntax improvements. Probably to keep git blame intact and avoid introducing new bugs.
Honestly, this reads as an obsession with theoretical concerns.
Rust has very arcane syntax and a lot of rules that developers coming from interpreted / garbage-collected languages (like the ones using these tools) would have a hard time grasping. It’s easy for people who are already familiar with it, but isn’t that always the case?
I'm saying the opposite, actually. I found it easier to contribute to Rust projects (with zero hours Rust experience, and much Python/Java/C/C++/Ada) because Rust projects are significantly easier to build. (Just a `cargo build` in the default case.)
An LSP is performance-critical code. It directly affects responsiveness of your IDE, or even the viable scope of a project that the LSP can handle.
Rust is both CPU- and memory-efficient, quite unlike Python. (It could have been OCaml / Reason, or Haskell, they are both reasonably fast and efficient, and very convenient to write stuff like typecheckers in. But the circle of possible contributors would be much narrower.)
It makes it easy to find performant and quality software by searching for "[insert tool description] rust", I personally don't mind! Seeing how 95% of the tools I use on the daily are written in Rust, I love finding new ones and am rarely disappointed.
> I'd prefer to have non-performance critical code for Python written in Python
A type checker is performance critical code. You can watch how Pylint, just a linter, written in Python, lints your source code line by line. It's so slow it can take 30 seconds to update the linting after you change some lines.
I feel like the likelihood that a project will say what language it is written in is much higher if that language is Rust. I like Rust but I do find this trend a little annoying. (Even though I acknowledge that "written in Rust" probably means the tool is relatively new, not buggy, and easy to use.)
I'm curious to know more about the Pyre to Pyrefly transition, specifically the rewrite in Rust. Was that merely a case of trading in a lesser-known language for the language du jour? Were there specific advantages they wanted to get out of Rust?
> Not only is Pyrefly written in a new language (Rust instead of OCaml), but its design deviates in a major way from Pyre.
I'm sure you had reasons to do it this way. But given sufficient time to market, implementing the algorithm in pyre and then tooling/llm assisted conversion to pyrefly would've been preferable.
May be you'd have had some humans in the loop initially. But that tech is getting better and aligned with the direction Meta and the rest of the industry are taking.
pyrefly is not tied to vscode? Also please try to be more considerate of people preferences, and pycharm is not strictly better. Remote dev on vscode is very convenient for me, should I go on the Internet saying that pycharm is trash? No
I’m not saying VS Code is trash but I think it’s closer to a text editor than an IDE. I even use it for some things non python but I remain curious to the fact why people use it for python.
It might not be tied to VS Code but the title clearly says “New […] IDE experience for…” which is why I commented. I had hoped to see something for PyCharm or even a new IDE.
I don't know the differences between the two well enough to know if it was the case here, but in my experience sometimes you need to innovate on a fork, or from scratch in order to create the space/freedom to do so.
Once a project is popular, it's harder to justify and be confident about major changes (aka https://m.xkcd.com/1172/)
It seems like the share a lot of the same goals but my impression is Poetry is much slower to pick up on standards. It’s normal to use uv with a project now that doesn’t have any [tool.uv] section in pyproject.toml at all but every poetry project I’ve seen is littered with [poetry] sections, even dependencies. Makes me not want to use it
I think astral and meta were both working on their own type-checkers independently. My current understanding is that meta released so they could preempt the initial release of ty. It seems like they're a bit further ahead in development. Not sure if there are going to be any real differences between the two down the line.
Sure, but in this case they are both implementations of a spec defined by PEPs, so a bit more like gcc vs clang (less tightly bound than those, of course, in design decisions). Neither company is trying to invent a new language here.
To repeat an earlier comment of mine from the launch of uv on hn (tl; dr: these new type checkers never support django):
The way these type checkers get fast is usually by not supporting the crazy rich reality of realworld python code.
The reason we're stuck on mypy at work is because it's the only type checker that has a plugin for Django that properly manages to type check its crazy runtime generated methods.
I wish more python tooling took the TS approach of "what's in the wild IS the language", as opposed to a "we only typecheck the constructs we think you SHOULD be using".
1. Maybe it's time to drop the crazy runtime generation and have something statically discoverable, or at least a way to annotate the typing statically.
2. Astral indicated already they plan to just add direct support for Django and other popular languages.
3. As people replied to similar comments on the previous threads (maybe to you?): that's not why ty is fast and why mypy is slow. It's also an easy claim to disprove: run both without plugins and you'll see ty is still 100x+ faster.
> 1. Maybe it's time to drop the crazy runtime generation and have something statically discoverable, or at least a way to annotate the typing statically.
That, and duck typing, are one of the biggest things that make Python what it is. If I have to drop all that for type checking and rewrite my code, why would I rewrite it in Python?
Having used Python for many years, it’s the least interesting aspect of the language. Almost all such tricks can be done with compile time meta programming, often even without API changes.
TS has the luxury of being its own distinct language, defining its own semantics and having its own compiler. You could have something like that targeting Python.
The only type checker that fully works (meaning it successfully performs the necessary type inference all for inherited objects) for our large and highly modular python codebase, is Pycharm (I'm guessing it's their own custom tool from the ground up? Not really sure, actually.)
So watch out, or this will become like Atom or Flow, an internal competitor of a technology that is surpassed by the more popular external open source version, leaving the directors/vps muttering to themselves "It's too bad that this team exists at all. Could we get rid of them and just switch to the open source stuff?"
Perhaps just something for the manager (Aaron Pollack?) to keep an eye on....
reply