Iceberg working with forks - can it be easier?

On 16 Feb 2019, at 06:19, Ben Coman <[hidden email]> wrote:

On Sat, 16 Feb 2019 at 02:06, Tim Mackinnon <[hidden email]> wrote:

Hi guys - I’ve spent a few hours scratching my head trying to understand why some of my Pull Requests to a project I had forked kept showing my previous commits when I thought I was all caught up.

It suddenly dawned on me, that when I had forked, and then done some work and then submitted a PR, and then applied it upstream that my fork is now no longer in sync with its upstream counterpart.

Having not done this in ages, it took me a while to then realise I have to do some git stuff to get it back in sync e.g. (and I think I’ve got this right)

git fetch upstream (or whatever name you gave it)
Git checkout master
Git merge upstream/master

So I guess my question is - wouldn’t it be helpful if this was a command in Iceberg? It seems quite common to fork a project (I think this is still recommended for pharo itself isn’t it?) - and then at some point you need to catchup with that origin again? Or am I missing something? I guess lots of stuff can go wrong - but if it does - you’d still get the same problems on the command line. It just seems that for normal situations - it would be handy to run this straight in Pharo.

I'm not fully-conversant with Iceberg, but if I guess right...

in Iceberg, right-click a repo and open its "Repository" window.

In the top-right click the "Add remote" button, to add the upstream repo.

That button is a bit hidden there.  It might be nice to have it as a menu item on "Remotes" in the first pane.

Then click the <Merge> button, then select the upstream master branch and you should be up to date.

cheers -ben

On 18 Feb 2019, at 08:56, Guillermo Polito <[hidden email]> wrote:

Hi!

On Fri, Feb 15, 2019 at 7:06 PM Tim Mackinnon <[hidden email]> wrote:

Hi guys - I’ve spent a few hours scratching my head trying to understand why some of my Pull Requests to a project I had forked kept showing my previous commits when I thought I was all caught up.

It suddenly dawned on me, that when I had forked, and then done some work and then submitted a PR, and then applied it upstream that my fork is now no longer in sync with its upstream counterpart.

Having not done this in ages, it took me a while to then realise I have to do some git stuff to get it back in sync e.g. (and I think I’ve got this right)

git fetch upstream (or whatever name you gave it)
Git checkout master
Git merge upstream/master

When you did your clone, did you clone the original repository or your fork's?

One way to simplify this workflow is:

 - you clone always the original repository

 - you create a branch and work there

 - then you'll push your branch to your fork + fork

That way your branch will be always up to date with the original repository, and no need to sync (actually the sync is done automatically when you push to your fork).

Of course this assumes you're doing a new clone every time you work :).

 

So I guess my question is - wouldn’t it be helpful if this was a command in Iceberg?

It would, there are several places where we could simplify the workflow for common cases, I agree.

 

It seems quite common to fork a project (I think this is still recommended for pharo itself isn’t it?)

I'd say its mandatory :). Otherwise you will not be able to push directly to it because of permission problems (actually all main dev branches are now blocked so **nobody**, not even admins, push to them directly).

 

- and then at some point you need to catchup with that origin again? Or am I missing something?

Well, it really depends on the workflow you use. Above I've explained a workflow that works when you clone every time.

Git is so complex and so low level that you can do lots of different things with it.

First thing is to understand what "catch up with origin" means. In my head it only means:

 - checkout your "to-catchup" branch

 - merge the remote "to-catchup" branch and merge it

All the rest is workflow dependent. Even (!!) I would argue that merging is not, strictly speaking, a correct way to catch up: If you have made changes into "to-catchup", then merging will not make both branches the same, but your "to-catchup" branch may have extra commits that you did not want to introduce...

And here I'm not saying you should not merge, actually I do it most of the times because I know that 90% of the times it will be ok, and I do rollback when I realize I was in the other 10%...

 

I guess lots of stuff can go wrong - but if it does - you’d still get the same problems on the command line.

Yes, and no. The "lots of stuff can go wrong" can be translated in most of the cases to "I had a merge conflict". But this is not git, this is git+pharo.

The thing is that with Git, your code is dead in files. With Iceberg, your code is in the files but it also may be alive in your image too. So updating code for what you have existing instances, or announcement subscriptions or other could (and probably will) mess up with the running code...

So to the typical merge conflict, you may add breaking your running system.

It just seems that for normal situations - it would be handy to run this straight in Pharo.

Thoughts from the git experts?

For these cases, you can do exactly as the command line from Iceberg's UI

 - change your branch to your "to-catchup" master

 - fetch

 - now you can go to the merge option, and choose to merge your remote branch into your checked-out branch.

 - and push

Of course, this simple workflow works most of the times, but you may find edge cases.

For those edge cases (typically updating iceberg itself, or trying to update pharo itself), in the checkout preview you will have the combo with the checkout strategy: choose the expert mode "do not load the code". This option will change the branch on git's world, but it will not load the code in your image.

That is useful if you have living instances for example.

This option will though make a diff between the checked out version and your image code and show you differences. (think this as a git reset --soft)

To me the fact that we are manipulating live code loaded in the image and not dead files makes a huge difference, and this makes that extrapolating git commands to iceberg is not so straight forward :). Also, understanding that git is stateful/context dependant, and applying an operation means changing your current state (modify your current branch, change your current branch) is useful to see the consequences.

In any case, any more concrete suggestions for enhancing this workflow are welcome. I think that neither Esteban, Pablo, me nor other contributors are skilled UI designers, and thinking about the best way to present a usecase to a user is super super hard :).

Guille, from his vacations

On 19 Feb 2019, at 15:45, Tim Mackinnon <[hidden email]> wrote:

Once again - super helpful - I’ll definitely bookmark this one. Half the problem is that every time I think I understand what is going on and confidently do a bunch of stuff I then discover that its not what I thought. I then jump over to IntelliJ to visualise what the current state is and then try understand how I got to that place - which hasn’t been from doing anything knowingly funky (that comes afterwards when I’ve tried to rationalise/rectify what I think I’m seeing).

If you don’t mind, I will try and repeat what you’ve said and keep a journal of what I’m doing so we can hopefully make sense of what has happened 3 times now.

So back to original setup - in GitHub you fork the main repo (pharo for you, but for me it was exercism/pharo due to the name issue - but actually this has proved a useful learning exercise).

So having forked from exercism/pharo to macta/pharo-exercism , in Pharo I go to clone my repo. (I’ve actually just removed my previous project and said to delete from file system as well)

So here’s the first thing - if I clone my fork - Pharo helpfully creates 2 remotes for me (exercism & origin) if I clone just the origin I then have to add the remote myself - and the prompt for that that is user and url (so a bit less friendly than having the extra GitHub option of user and project).

So for now - I’ll stick with what I’ve done before (but do you think doing it the more awkward way is better? And can we make it less awkward if it is).

Now just to be sure - I’ll make sure my fork and origin are in sync (and this is where now I notice why it might have been going wrong, because my current local master is based on that of my fork - which I notice currently on GitHub says its “10 commits ahead of exercism:master”) - and this might be why I’m seeing all those extra commits in my PR (?). Scratching my head on this I’ve realised that when I first started out - I accidentally committed to master on my fork before realising that I had forgotten to create a branch - which I then did (but I guess I left my fork master in a bad state).

So I have now done:

git reset --hard exercism/master

Now my GitHub fork page looks like its correct.

So I’ve done some categorisation changes in a branch and done a commit and pushed those changes to my fork - but when I go to generate a PR on GitHub in my upstream - its showing 10 commits… grrrr

So it looks like this (and yes its a mess because I’ve been struggling with this for day and getting the same sort of problem)

<PastedGraphic-1.png>

The only thing I can think is that after doing that reset - I can’t recall if I did a pull in pharo - so possibly my image was still synced to the repo prior to me making it equal to upstream?

But I’ve now just switched to master in my image - created a new branch, and made a simple comment change and pushed that branch back to my fork and its still showing 11 commits (when there should be just 1).

So I don’t understand whats going on?

My upstream is exercism/pharo-smalltalk and my fork is macta/pharo-exercism - the comment change I just made was id: 84e581c38d351baddf98a172a01b5bf475ec2e25.

Is it possible my image is just in a funny state and that simply removing the project at the beginning wasn’t enough?

Tim

On 19 Feb 2019, at 13:08, Guillermo Polito <[hidden email]> wrote:

On Tue, Feb 19, 2019 at 2:08 AM Tim Mackinnon <[hidden email]> wrote:

Hi Guille - thanks for taking the time to write this up… in the middle there I finally spotted the crucial bit “clone the original, push to your fork”. I think thats what I’ve been doing wrong - and it leads to all kinds of confusion (at least I’m hoping this is what it is).

You've been doing nothing wrong :).

Both are valid things in Git, and to talk about rights and wrongs are fuzzy at the least there.

 

As I have cloned from my Fork (and in my defence - its because we were given a repository that collides with the pharo one - exercism/pharo - and P7 doesn’t let us clone from that) - so I’ve had to clone from my fork.

Yeh, that's a bug. It should not be difficult to fix.

The thing is, iceberg distinguishes repositories/projects by name.

And names are obtained from the repository/project directory name.

Then, the plugin made to simplify contributions to pharo, detects the repositories called "pharo" as pharo themselves and do some extra configuration. What happens in your case is that your repository suffers from that extra configuration, which is not the right thing to do in your case...

One Idea that was poking around in my head is to be able to "rename" a project independently of its location in disk.

We already have a metadata file at the root of the repository. We could add in that metadata file properties like:

 - the project's name (for example you could call it "pharo-exercism" in there, while keeping the tree structure)

 - an extra special property "isPharoRepository : true" that may be defined only for pharo

 - a canonical-repository url which could be used to compare against pharo's canonical url

Implementing the detection of pharo's repository using any of those strategies could solve your problem...

What do you think?

 

It all seemed to work until I started submitting my PR’s and then they seem to include commits going much further back than I think they should need Even wen I thought I was being careful and syncing to master - it still seems to go back to some of my earliest work.

This looks like a mis-manipulation of branches, and It smells to me that it is not related to the issue above...

Next time it happens, I'd like, if you can, that you share with me the exact:

 - branch

 - commitish

that you're trying to make a PR from, to see if we can see the real cause, and at least build some docs on it ^^.

Now that we have managed to get the repo renamed - I will try working like you have suggested and make sure that it is working like you describe - as it shouldn’t be as complicated as this to make clean isolated changes and then submit them as a PR. (otherwise Pharo would fall apart).

One question I do have however - is what is the best procedure if you realised when making changes that 

A) you forgot to make changes in a branch (so are working on master)

It depends if your changes have been pushed or not.

1) First thing is to fix your local branch with the expression I told in a previous email

repo branch commit: repo branch commit parent.

2) If you have already pushed, that means that you have to rewrite the history on the remote side, and that would require a push force (from the command line so far, unless you would like to implement that in iceberg :P).

 

B) you are working in a branch but decide some of your changes should go in another branch itself based on master (how to stash?)

There is no stash mechanism in iceberg (so far).

What I do is:

 - I checkout the base branch (master in this case) in expert mode => do not touch my image code, then the image itself will work as a stash

 - I now checkout a new branch (that will "fork" from the last commit in my master branch)

 - I commit only the things I want to commit.

(see that this is not much different from what you would have done from the terminal except that the checkout strategy will make the stashing unnecessary:

 - git stash

 - git checkout master

 - git checkout -b myBranch

 - git stash pop

 - git commit

)

Then you can either 

 - discard all your extra changes (I'd suggest to have committed them earlier :)) to be in a clean master+yournewchanges

 - or you can go back to your first branch

 

These are 2 common scenarios I’ve found when trying to be a good citizen for my little project. Do I have to go to the terminal to do this stuff -  or can I cleanly do these kinds of operations in Pharo (which I think you should be able to do, as its so easy to get into either of those states).

I think you can do everything so far except the push force.

Still I'm not happy with what we have (and I'd like to have 48h days) :)

To help in your first scenario:

- It should not be difficult to get the reset on the UI (https://github.com/pharo-vcs/iceberg/issues/1174) ;)

- The same with the push force

For the second scenario, what bothers me the most is the special checkout that may confuse people.

Also, while you're doing it, you may have not committed your code anywhere => and that's dangerous, because you may lose it.

So, imagine this: you are in branch A and you have these changes that you wan to put in branch B.

Instead of doing strange stashing/checkout blah, you create a new branch temp/AtoB and commit there.

And then you move to B and apply a smart cherry pick operation of your changes.

Martin was visiting the team last week and he had a super nice prototype: you select what you want to cherry pick from a branch and it automatically selects all (static) dependencies in the same branch, and proposes you doing a commit/merge of that.

I don't know how far are we from this, but it would be super nice to do easy backporting between branches (think pharo8 -> pharo7).

Guille

Thanks in advance.

Tim

On 18 Feb 2019, at 08:56, Guillermo Polito <[hidden email]> wrote:

Hi!

On Fri, Feb 15, 2019 at 7:06 PM Tim Mackinnon <[hidden email]> wrote:

Hi guys - I’ve spent a few hours scratching my head trying to understand why some of my Pull Requests to a project I had forked kept showing my previous commits when I thought I was all caught up.

It suddenly dawned on me, that when I had forked, and then done some work and then submitted a PR, and then applied it upstream that my fork is now no longer in sync with its upstream counterpart.

Having not done this in ages, it took me a while to then realise I have to do some git stuff to get it back in sync e.g. (and I think I’ve got this right)

git fetch upstream (or whatever name you gave it)
Git checkout master
Git merge upstream/master

When you did your clone, did you clone the original repository or your fork's?

One way to simplify this workflow is:

 - you clone always the original repository

 - you create a branch and work there

 - then you'll push your branch to your fork + fork

That way your branch will be always up to date with the original repository, and no need to sync (actually the sync is done automatically when you push to your fork).

Of course this assumes you're doing a new clone every time you work :).

 

So I guess my question is - wouldn’t it be helpful if this was a command in Iceberg?

It would, there are several places where we could simplify the workflow for common cases, I agree.

 

It seems quite common to fork a project (I think this is still recommended for pharo itself isn’t it?)

I'd say its mandatory :). Otherwise you will not be able to push directly to it because of permission problems (actually all main dev branches are now blocked so **nobody**, not even admins, push to them directly).

 

- and then at some point you need to catchup with that origin again? Or am I missing something?

Well, it really depends on the workflow you use. Above I've explained a workflow that works when you clone every time.

Git is so complex and so low level that you can do lots of different things with it.

First thing is to understand what "catch up with origin" means. In my head it only means:

 - checkout your "to-catchup" branch

 - merge the remote "to-catchup" branch and merge it

All the rest is workflow dependent. Even (!!) I would argue that merging is not, strictly speaking, a correct way to catch up: If you have made changes into "to-catchup", then merging will not make both branches the same, but your "to-catchup" branch may have extra commits that you did not want to introduce...

And here I'm not saying you should not merge, actually I do it most of the times because I know that 90% of the times it will be ok, and I do rollback when I realize I was in the other 10%...

 

I guess lots of stuff can go wrong - but if it does - you’d still get the same problems on the command line.

Yes, and no. The "lots of stuff can go wrong" can be translated in most of the cases to "I had a merge conflict". But this is not git, this is git+pharo.

The thing is that with Git, your code is dead in files. With Iceberg, your code is in the files but it also may be alive in your image too. So updating code for what you have existing instances, or announcement subscriptions or other could (and probably will) mess up with the running code...

So to the typical merge conflict, you may add breaking your running system.

It just seems that for normal situations - it would be handy to run this straight in Pharo.

Thoughts from the git experts?

For these cases, you can do exactly as the command line from Iceberg's UI

 - change your branch to your "to-catchup" master

 - fetch

 - now you can go to the merge option, and choose to merge your remote branch into your checked-out branch.

 - and push

Of course, this simple workflow works most of the times, but you may find edge cases.

For those edge cases (typically updating iceberg itself, or trying to update pharo itself), in the checkout preview you will have the combo with the checkout strategy: choose the expert mode "do not load the code". This option will change the branch on git's world, but it will not load the code in your image.

That is useful if you have living instances for example.

This option will though make a diff between the checked out version and your image code and show you differences. (think this as a git reset --soft)

To me the fact that we are manipulating live code loaded in the image and not dead files makes a huge difference, and this makes that extrapolating git commands to iceberg is not so straight forward :). Also, understanding that git is stateful/context dependant, and applying an operation means changing your current state (modify your current branch, change your current branch) is useful to see the consequences.

In any case, any more concrete suggestions for enhancing this workflow are welcome. I think that neither Esteban, Pablo, me nor other contributors are skilled UI designers, and thinking about the best way to present a usecase to a user is super super hard :).

Guille, from his vacations

-- 

Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr Web: http://guillep.github.io Phone: +33 06 52 70 66 13