[dart-misc] [dart-announce] Dart 2 intent to remove: the build and serve commands from pub

['Kevin Moore' via Dart Announcements]

*pub is getting out of the build system business and going back to its
roots: being an amazing package manager for the Dart ecosystem.In the next
few weeks, we plan to remove support for the build and serve commands from
pub. This change will happen in our regular dev channel releases.Don't
panic!All of the package management features will remain unchanged: get,
upgrade, run, publish, and friends will continue to work as you
expect.Replacing pub build and pub serveIf you use pub build and pub serve
to develop and deploy web applications, we've outlined the migration steps
in our Dart 2 Migration Guide <https://webdev.dartlang.org/dart-2>. The
documentation for package:build_runner
<https://webdev.dartlang.org/tools/build_runner> is also a good place to
start.Not ready to migrate off pub build and pub serve?That's okay. Your
workflow will not be affected if either of these is true: - You're on a
Dart 1.x release.- You don't upgrade to the latest Dart 2-dev until you
have updated your workflow to use build_runner.Where can I get help?Start
with the links provided. If you have questions or get stuck, post a
question on Stack Overflow <https://stackoverflow.com/tags/dart> or join us
on the dart-lang/build Gitter room <https://gitter.im/dart-lang/build>. If
you find an issue, please file it in the dart-lang/build repository
<https://github.com/dart-lang/build>.*
--
For more ways to connect visit https://www.dartlang.org/community
---
You received this message because you are subscribed to the Google Groups "Dart Announcements" group.
Visit this group at https://groups.google.com/a/dartlang.org/group/announce/.

['Kevin Moore' via Dart Misc]

FYI:

Nate just posted a write-up with details about our build system
evolution: http://lessworsemorebetter.com/2018/04/17/dart-builds-how-did-we-get-here.html

Almost in the same breath, he also landed the pull request
<https://github.com/dart-lang/pub/pull/1871> that removes build/serve from
pub. (Points for naming the branch "kill-bill".)

To follow the overall progress for this change in the SDK, subscribe
to https://github.com/dart-lang/sdk/issues/32593

On Monday, April 9, 2018 at 5:07:04 PM UTC-7, 'Kevin Moore' via Dart

Post by 'Kevin Moore' via Dart Announcements
*pub is getting out of the build system business and going back to its
roots: being an amazing package manager for the Dart ecosystem.In the next
few weeks, we plan to remove support for the build and serve commands from
pub. This change will happen in our regular dev channel releases.Don't
panic!All of the package management features will remain unchanged: get,
upgrade, run, publish, and friends will continue to work as you
expect.Replacing pub build and pub serveIf you use pub build and pub serve
to develop and deploy web applications, we've outlined the migration steps
in our Dart 2 Migration Guide <https://webdev.dartlang.org/dart-2>. The
documentation for package:build_runner
<https://webdev.dartlang.org/tools/build_runner> is also a good place to
start.Not ready to migrate off pub build and pub serve?That's okay. Your
workflow will not be affected if either of these is true: - You're on a
Dart 1.x release.- You don't upgrade to the latest Dart 2-dev until you
have updated your workflow to use build_runner.Where can I get help?Start
with the links provided. If you have questions or get stuck, post a
question on Stack Overflow <https://stackoverflow.com/tags/dart> or join us
on the dart-lang/build Gitter room <https://gitter.im/dart-lang/build>. If
you find an issue, please file it in the dart-lang/build repository
<https://github.com/dart-lang/build>.*
--
For more ways to connect visit https://www.dartlang.org/community
---
You received this message because you are subscribed to the Google Groups
"Dart Announcements" group.
Visit this group at
https://groups.google.com/a/dartlang.org/group/announce/.

[tatumizer-v0.2]

A bit off topic: I'd like to see some abstract mathematical definition of
the problem the build is trying to solve. I tried to google it, but either
my keywords were wrong, or said definition simply doesn't exist.
I mean, a definition in terms of graphs, or constraint solvers, or
something like that. The very fact that we have race conditions during the
build (mentioned in Nate's article) strongly suggests that the build isn't
fully aware of what problem it's trying to solve, and instead just *doing
something* towards an unknown goal :)
My conjecture is that as soon as the problem gets defined, it will turn out
that the only way to solve it is to honestly write a program (procedural
one, in a Turing-complete language) that does it for a given specific case.
This would make any shortcuts via declarative syntax impossible in
principle. I'm not sure though.

['Bob Nystrom' via Dart Misc]

Barback can get itself into weird race conditions because of the
combination of a few things:


1. *Cyclic dependencies between packages are allowed.* Dart the language
has always allowed cyclic imports and they are widely used in practice.
Because of that, we felt it import that pub allow cyclic package
dependencies. Cycles at that granularity aren't in common, but they do
happen, especially around dev dependencies. For example, the test package
uses args for its arg parsing, and args uses test for its tests.

2. *The set of inputs and outputs for a transformer is imperatively
determined.* Some build systems like Bazel require you to explicitly
declaratively list all of the inputs and outputs of each build step. This
lets you construct the entire build graph before performing any work, which
is nice. The downside is that it can be verbose, tedious, and error prone
to write all of those down. Consider a transformer that takes a folder full
of PNG files and produces one big sprite sheet out of all of them. In
barback, you can write a transformer that imperatively walks the input
directory, discovers the files, and lists them as inputs. Drop a new PNG in
there and it automatically gets picked up.

Barback was designed for relatively simple dependency graphs with lots
of small assets — things like spriting images, transpiling CSS, etc. so it
optimized for ease-of-use over untangling giant dependency graphs. But,
since Dart has no other metaprogramming story, barback ended up being the
de facto solution for things like Angular template compilation and other
much more complex build scenarios that it wasn't intended for.

3. *A transformer can change a file "in place".* Barback allows you to
produce an output file whose path is the same as a previous input file. We
felt this was important to allow transforming files that are referenced by
other files. Consider a transformer that, say, compiles async/await in Dart
files to vanilla future code. If we don't allow transformers to overwrite
files, then the transformer has to output that file to some new path. But
all of the other Dart files importing that one are still importing the old
path and won't pick up the transformed one. So you'd have to transform
those too to fix their imports. But then *those* files need new paths,
so you need to transform everything that imports *them*. The result is
you have to transform every single Dart file just to make a change to a
single one.

When you put these together, you can get into this situation:

1. Package a reads an input from package b.

2. We need to wait for b's transformers to finish running before that
input can be provided because we don't know if it will be produced (rule 2
above) or modified (rule 3) by some transformer in b.

3. But package b also reads an input from package a (rule 1), so we need
to wait for *its* transformers to run.

4. Deadlock.

I'm not an expert on the new build stuff, but I believe it addresses these
like so:

1. I don't think it allows dependency cycles between packages at all. Or
it may allow them between packages but not between build targets, which are
fine-grained than entire packages.

2. Inputs and outputs are explicitly listed, though I think some tooling
is provided to auto-generate these manifests for common cases.

3. You can't output to a path where a file already exists. No
"overwriting". Instead, all build steps produce new files at new paths.
This means those outputs can be written to disc. Thus, to solve the "modify
a single file without breaking references to it" problem, you write your
imports to point to the *output* path not the *original* path. Since the
build outputs will be on disc, tools like IDEs can find them. It's a little
weird because it means when authoring code, you need to be more aware of
what the build system outputs where, but it simplifies a lot of other
problems.

Cheers!

– bob

Post by tatumizer-v0.2
A bit off topic: I'd like to see some abstract mathematical definition of
the problem the build is trying to solve. I tried to google it, but either
my keywords were wrong, or said definition simply doesn't exist.
I mean, a definition in terms of graphs, or constraint solvers, or
something like that. The very fact that we have race conditions during the
build (mentioned in Nate's article) strongly suggests that the build isn't
fully aware of what problem it's trying to solve, and instead just *doing
something* towards an unknown goal :)
My conjecture is that as soon as the problem gets defined, it will turn
out that the only way to solve it is to honestly write a program
(procedural one, in a Turing-complete language) that does it for a given
specific case. This would make any shortcuts via declarative syntax
impossible in principle. I'm not sure though.
--
For more ways to connect visit https://www.dartlang.org/community
---
You received this message because you are subscribed to the Google Groups
"Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit https://groups.google.com/a/
dartlang.org/d/msgid/misc/24ab6f3a-2a68-4603-a927-
58c011f66c1f%40dartlang.org
<https://groups.google.com/a/dartlang.org/d/msgid/misc/24ab6f3a-2a68-4603-a927-58c011f66c1f%40dartlang.org?utm_medium=email&utm_source=footer>
.

['Jake Macdonald' via Dart Misc]

Post by 'Bob Nystrom' via Dart Misc
1. I don't think it allows dependency cycles between packages at all.
Or it may allow them between packages but not between build targets, which
are fine-grained than entire packages.
It does allow package cycles, but it essentially tries to treat them like

"one" package as much as possible.

Essentially, for any package which is not in a cycle we order the phases
such that all its dependencies are built in their entirety first. However,
if there is a package cycle then we choose a deterministic ordering between
the two packages, and apply each builder one at a time to each package.

As an example, if there is a cycle between package "a" and package "b", and
they each have the angular, summary, and ddc builders applied, then we
would add conceptually these phases:

1. angular on package:a
2. angular on package:b
3. summaries on package:a
4. summaries on package:b
5. ddc on package:a
6. ddc on package:b

If they were not in a cycle, say "a" depends on "b" but not the other way
around, the ordering would look like:

1. angular on package:b
2. summaries on package:b
3. ddc on package:b
4. angular on package:a
5. summaries on package:a
6. ddc on package:a

This "just works, in all existing real world scenarios we know of, although
you could certainly devise a scenario where it wouldn't do what you wanted
if you really tried. For that case the new build system also allows you to
create a custom build script if the automatic ordering doesn't work for
you, where you can choose the exact ordering of all the builders.

Post by 'Bob Nystrom' via Dart Misc
1. Inputs and outputs are explicitly listed, though I think some
tooling is provided to auto-generate these manifests for common cases.
Yes builder authors have to declare their mapping of input files to output

files. This allows us to build the graph ahead of time.

['Nate Bosch' via Dart Misc]

bcc ***@dartlang.org

On Thu, Apr 19, 2018 at 9:36 AM 'Bob Nystrom' via Dart Misc <

Post by 'Bob Nystrom' via Dart Misc
Barback can get itself into weird race conditions because of the
1. *Cyclic dependencies between packages are allowed.* Dart the
language has always allowed cyclic imports and they are widely used in
practice. Because of that, we felt it import that pub allow cyclic package
dependencies. Cycles at that granularity aren't in common, but they do
happen, especially around dev dependencies. For example, the test package
uses args for its arg parsing, and args uses test for its tests.
2. *The set of inputs and outputs for a transformer is imperatively
determined.* Some build systems like Bazel require you to explicitly
declaratively list all of the inputs and outputs of each build step. This
lets you construct the entire build graph before performing any work, which
is nice. The downside is that it can be verbose, tedious, and error prone
to write all of those down. Consider a transformer that takes a folder full
of PNG files and produces one big sprite sheet out of all of them. In
barback, you can write a transformer that imperatively walks the input
directory, discovers the files, and lists them as inputs. Drop a new PNG in
there and it automatically gets picked up.
Barback was designed for relatively simple dependency graphs with lots
of small assets — things like spriting images, transpiling CSS, etc. so it
optimized for ease-of-use over untangling giant dependency graphs. But,
since Dart has no other metaprogramming story, barback ended up being the
de facto solution for things like Angular template compilation and other
much more complex build scenarios that it wasn't intended for.
3. *A transformer can change a file "in place".* Barback allows you to
produce an output file whose path is the same as a previous input file. We
felt this was important to allow transforming files that are referenced by
other files. Consider a transformer that, say, compiles async/await in Dart
files to vanilla future code. If we don't allow transformers to overwrite
files, then the transformer has to output that file to some new path. But
all of the other Dart files importing that one are still importing the old
path and won't pick up the transformed one. So you'd have to transform
those too to fix their imports. But then *those* files need new paths,
so you need to transform everything that imports *them*. The result is
you have to transform every single Dart file just to make a change to a
single one.
1. Package a reads an input from package b.
2. We need to wait for b's transformers to finish running before that
input can be provided because we don't know if it will be produced (rule 2
above) or modified (rule 3) by some transformer in b.
3. But package b also reads an input from package a (rule 1), so we
need to wait for *its* transformers to run.
4. Deadlock.
I'm not an expert on the new build stuff, but I believe it addresses these
1. I don't think it allows dependency cycles between packages at all.
Or it may allow them between packages but not between build targets, which
are fine-grained than entire packages.

We really _wanted_ to disallow dependency cycles but we found it wasn't
practical in our current ecosystem. If and when we revisit bazel support we
will need to disallow target-level cycles. Today build_runner allows either
and we interleave builders across them so there is a supported solution to
"I need information from other sources in a cycle" which is to split up
your builder into multiple steps.

Post by 'Bob Nystrom' via Dart Misc
2. Inputs and outputs are explicitly listed, though I think some
tooling is provided to auto-generate these manifests for common cases.

Outputs are all determined by file extensions, inputs are allowed
dynamically. The important thing is that the set of possible outputs is
static.

Post by 'Bob Nystrom' via Dart Misc
3. You can't output to a path where a file already exists. No
"overwriting". Instead, all build steps produce new files at new paths.
This means those outputs can be written to disc. Thus, to solve the "modify
a single file without breaking references to it" problem, you write your
imports to point to the *output* path not the *original* path. Since the
build outputs will be on disc, tools like IDEs can find them. It's a little
weird because it means when authoring code, you need to be more aware of
what the build system outputs where, but it simplifies a lot of other
problems.

Cheers!

Post by 'Bob Nystrom' via Dart Misc
– bob

Post by tatumizer-v0.2
A bit off topic: I'd like to see some abstract mathematical definition
of the problem the build is trying to solve. I tried to google it, but
either my keywords were wrong, or said definition simply doesn't exist.
I mean, a definition in terms of graphs, or constraint solvers, or
something like that. The very fact that we have race conditions during the
build (mentioned in Nate's article) strongly suggests that the build isn't
fully aware of what problem it's trying to solve, and instead just *doing
something* towards an unknown goal :)
My conjecture is that as soon as the problem gets defined, it will turn
out that the only way to solve it is to honestly write a program
(procedural one, in a Turing-complete language) that does it for a given
specific case. This would make any shortcuts via declarative syntax
impossible in principle. I'm not sure though.
--
For more ways to connect visit https://www.dartlang.org/community
---
You received this message because you are subscribed to the Google Groups
"Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/a/dartlang.org/d/msgid/misc/24ab6f3a-2a68-4603-a927-58c011f66c1f%40dartlang.org
<https://groups.google.com/a/dartlang.org/d/msgid/misc/24ab6f3a-2a68-4603-a927-58c011f66c1f%40dartlang.org?utm_medium=email&utm_source=footer>
.

--
For more ways to connect visit https://www.dartlang.org/community
---
You received this message because you are subscribed to the Google Groups
"Dart Misc" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/a/dartlang.org/d/msgid/misc/CAF8T8Lz6ORU1a6XGfc8YxJpaefX0Rs%2Bx%3DKG%2BaezAXAiMtcWRTg%40mail.gmail.com
<https://groups.google.com/a/dartlang.org/d/msgid/misc/CAF8T8Lz6ORU1a6XGfc8YxJpaefX0Rs%2Bx%3DKG%2BaezAXAiMtcWRTg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

['Kevin Moore' via Dart Announcements]

Dart SDK 2.0.0-dev.51.0 has just been released with this change.

Post by 'Kevin Moore' via Dart Announcements
*pub is getting out of the build system business and going back to its
roots: being an amazing package manager for the Dart ecosystem.In the next
few weeks, we plan to remove support for the build and serve commands from
pub. This change will happen in our regular dev channel releases.Don't
panic!All of the package management features will remain unchanged: get,
upgrade, run, publish, and friends will continue to work as you
expect.Replacing pub build and pub serveIf you use pub build and pub serve
to develop and deploy web applications, we've outlined the migration steps
in our Dart 2 Migration Guide <https://webdev.dartlang.org/dart-2>. The
documentation for package:build_runner
<https://webdev.dartlang.org/tools/build_runner> is also a good place to
start.Not ready to migrate off pub build and pub serve?That's okay. Your
workflow will not be affected if either of these is true: - You're on a
Dart 1.x release.- You don't upgrade to the latest Dart 2-dev until you
have updated your workflow to use build_runner.Where can I get help?Start
with the links provided. If you have questions or get stuck, post a
question on Stack Overflow <https://stackoverflow.com/tags/dart> or join us
on the dart-lang/build Gitter room <https://gitter.im/dart-lang/build>. If
you find an issue, please file it in the dart-lang/build repository
<https://github.com/dart-lang/build>.*

--
For more ways to connect visit https://www.dartlang.org/community
---
You received this message because you are subscribed to the Google Groups "Dart Announcements" group.
Visit this group at https://groups.google.com/a/dartlang.org/group/announce/.