Difference between revisions of "First Steps with Gitea"

From SME Server
Jump to navigationJump to search
(Added CURL to create the repo and also conform contents to Trevor's organisation.)
Line 4: Line 4:
 
# $1  = Module name e.g. smeserver-ddclient
 
# $1  = Module name e.g. smeserver-ddclient
 
# $2 = "contrib" ...anything else -=> core module
 
# $2 = "contrib" ...anything else -=> core module
 +
if [ $1 = "" ]
 +
then
 +
echo "cvs2git <modulename> <contrib|base>"
 +
exit 0
 +
fi
 
LocalUser="brianr"
 
LocalUser="brianr"
 
RemoteUser="brianr"
 
RemoteUser="brianr"
Line 25: Line 30:
 
cd $1/contribs10
 
cd $1/contribs10
 
make prep
 
make prep
 +
 +
#Extract the directory name for the file tree (it will include the name of the module plus some version information)
 +
cd $CVSFiles/$1/contribs10/
 +
TREEPATH=`ls -d */ | grep $1 | grep -v "\.old" | grep -v "_sme"`
 +
echo "TreePATH:$TREEPATH"
  
 
# Create the local Git repository
 
# Create the local Git repository
Line 31: Line 41:
 
git init $1
 
git init $1
 
cd $1
 
cd $1
−
echo `pwd`
+
echo "PWD:"`pwd`
 
#pull in all the files we want
 
#pull in all the files we want
 
if [ $2 == "contrib" ]; then
 
if [ $2 == "contrib" ]; then
−
    mkdir -p contribs10
+
     for fpath in "$TREEPATH" "*.spec" "Makefile" "$TREEPATH/createlinks"
−
     for fpath in "*.patch" "*.spec" "*.gz" "*.tgz" "*.xz" "Makefile"  
 
 
     do  
 
     do  
−
     cp $CVSFiles/$1/contribs10/$fpath ./contribs10/
+
     cp -R $CVSFiles/$1/contribs10/$fpath .
 
     done
 
     done
 +
    mv $TREEPATH/root ./root
 +
    rm -f $TREEPATH/*
 +
    rmdir $TREEPATH
 
else
 
else
−
mkdir -p sme10
+
     for fpath in "$TREEPATH" "*.spec" "Makefile" "$TREEPATH/createlinks"
−
     for fpath in "*.patch" "*.spec" "*.gz" "*.tgz" "*.xz" "Makefile"  
 
 
     do  
 
     do  
−
        cp $CVSFiles/$1/sme10/$fpath ./sme10/
+
    cp -R $CVSFiles/$1/sme10/$fpath .
 
     done
 
     done
 +
    mv $TREEPATH/root ./root
 +
    rm -f $TREEPATH/*
 +
    rmdir $TREEPATH
 
fi
 
fi
  
−
#Get Clone the common directory
+
#Now edit the spec file to remove all those patches and adjust the paths
 +
sed -i '/^Patch/d; /^%patch/d' $1.spec
 +
# no paths adjusted yet!
 +
 
 +
#Create the tar file needed (currently) for the rpm build
 +
tar -czf $TREEPATH.tar.gz root/
 +
 
 +
#Clone the common directory
 
git clone https://src.koozali.org/brianr/common.git
 
git clone https://src.koozali.org/brianr/common.git
 
# and delete the .git else it pushes back a link to the repo
 
# and delete the .git else it pushes back a link to the repo
Line 53: Line 74:
 
rm -fr ".git"
 
rm -fr ".git"
 
cd ..
 
cd ..
 +
 +
#
 +
# Use the https interface to create the remote repo (it will just bounce back if it exists already)
 +
# Currently this creates a private repo (making it public here causes a gitea internal server error, as does auto_init
 +
# Note that the token decides where the repo is created (i.e. under what user)
 +
# We will need this to be "organisation" (smecontribs or smeserver) finally.
 +
#
 +
curl -X 'POST'  'https://src.koozali.org/api/v1/user/repos'  \
 +
-H 'accept: application/json'  \
 +
-H 'Authorization: token 28b6ebca17d6a70d6c1848e8edce4007f072961c'  \
 +
-H 'Content-Type: application/json'  \
 +
-d '{
 +
  "auto_init": false,
 +
  "default_branch": "master",
 +
  "description": "'"SMEServer git repo for $1 $2"'",
 +
  "gitignores": "",
 +
  "issue_labels": "",
 +
  "license": "Apache 2.0",
 +
  "name": "'"$1"'",
 +
  "private": true,
 +
  "readme": "",
 +
  "template": false,
 +
  "trust_model": "default"
 +
}'
 +
  
 
#stage and commit them
 
#stage and commit them
Line 66: Line 112:
 
echo "Created $1 Repo in $RemoteRepoURL"
 
echo "Created $1 Repo in $RemoteRepoURL"
 
exit 0  
 
exit 0  
 +
 +
 +
 
</syntaxhighlight>
 
</syntaxhighlight>
  

Revision as of 18:28, 15 February 2023

Initial script to take the latest version of a module and move it to Gitea.

#!/bin/bash
# $1  = Module name e.g. smeserver-ddclient
# $2 = "contrib" ...anything else -=> core module
if [ $1 = "" ]
then
	echo "cvs2git <modulename> <contrib|base>"
	exit 0
fi
LocalUser="brianr"
RemoteUser="brianr"
if [ $2 == "contrib" ]; then
  CVSFiles="/home/$LocalUser/smecontribs/rpms"
else 
  CVSFiles="/home/$LocalUser/smeserver/rpms"
fi
GitFiles="/home/$LocalUser/GitFiles"
RemoteRepoURL="https://src.koozali.org/$RemoteUser/$1"
mkdir -p $GitFiles

#Make sure credentials taken from store
git config --global credential.helper store #The first time you pull or push it will ask for credentials and then save them in a file.

# Clone the CVS repository
cd $CVSFiles
cvs -d:ext:shell.koozali.org:/cvs/smecontribs checkout $1

# Fully populate the directory (gets the tar file - which is not strictly in CVS)
cd $1/contribs10
make prep

#Extract the directory name for the file tree (it will include the name of the module plus some version information)
cd $CVSFiles/$1/contribs10/
TREEPATH=`ls -d */ | grep $1 | grep -v "\.old" | grep -v "_sme"`
echo "TreePATH:$TREEPATH"

# Create the local Git repository
cd $GitFiles
rm -rf $1
git init $1
cd $1
echo "PWD:"`pwd`
#pull in all the files we want
if [ $2 == "contrib" ]; then
    for fpath in "$TREEPATH" "*.spec"  "Makefile" "$TREEPATH/createlinks"
    do 
    	cp -R $CVSFiles/$1/contribs10/$fpath .
    done
    mv $TREEPATH/root ./root
    rm -f $TREEPATH/*
    rmdir $TREEPATH
else
    for fpath in "$TREEPATH" "*.spec"  "Makefile" "$TREEPATH/createlinks"
    do 
    	cp -R $CVSFiles/$1/sme10/$fpath .
    done
    mv $TREEPATH/root ./root
    rm -f $TREEPATH/*
    rmdir $TREEPATH
fi

#Now edit the spec file to remove all those patches and adjust the paths
sed -i '/^Patch/d; /^%patch/d' $1.spec
# no paths adjusted yet!

#Create the tar file needed (currently) for the rpm build
tar -czf $TREEPATH.tar.gz root/

#Clone the common directory
git clone https://src.koozali.org/brianr/common.git
# and delete the .git else it pushes back a link to the repo
cd common
rm -fr ".git"
cd ..

#
# Use the https interface to create the remote repo (it will just bounce back if it exists already)
# Currently this creates a private repo (making it public here causes a gitea internal server error, as does auto_init
# Note that the token decides where the repo is created (i.e. under what user)
# We will need this to be "organisation" (smecontribs or smeserver) finally.
#
curl -X 'POST'   'https://src.koozali.org/api/v1/user/repos'   \
-H 'accept: application/json'   \
-H 'Authorization: token 28b6ebca17d6a70d6c1848e8edce4007f072961c'   \
-H 'Content-Type: application/json'   \
-d '{
  "auto_init": false,
  "default_branch": "master",
  "description": "'"SMEServer git repo for $1 $2"'",
  "gitignores": "",
  "issue_labels": "",
  "license": "Apache 2.0",
  "name": "'"$1"'",
  "private": true,
  "readme": "",
  "template": false,
  "trust_model": "default"
}'


#stage and commit them
git add -A
git commit -m "initial commit of file from CVS for $1"

# Create the remote repo and push the converted git repository to the remote
git remote remove origin
git remote add origin $RemoteRepoURL
# Need to have created the repo at this point....
git pull --no-edit origin master
git push -u origin master
echo "Created $1 Repo in $RemoteRepoURL"
exit 0

Issues:

  1. How to partition the repos between smecontribs and core? different User? - smecontribs and smeserver directories? Then module name.With contribs10 / sme10 directories underneath, and also the common directory.
  2. Need ssh access to gitea for automatic creation of repos, else each one has to be created by hand!
  3. Realising that I need the contribs10/11/12 directories under the module name if I am going to get make mockbuild to work.
  4. Are going to hold the rpm's in the git repo as well? Otherwise we need the .gitignore set to *.rpm
  5. I presume we will hold the file tree tar file in git?
  6. I also presume that we will have one git repo per module? Rather than share Importing the a repo across all modules?


Bringing across all the CVS history as well...

According to google, git has a parameter "cvsimport" since version 2.1, however the version in my Rocky 8 system is 2.3.1, but has not such parameter. Latest comments I can find are for 2012, and some say it is not very reliable, so I suspect it has been withdrawn.

Only tool I can find is "cvs-fast-import" which will work with git-fast-import to import all the CVS data (inc history)

No rpms seem to exist, so I am having to build it on the Rocky 8 system (on which I also have CVS installed, and have access to the SME CVS)

Code for cvs-fast-import is here : https://gitlab.com/esr/cvs-fast-export

I needed 

git clone https://gitlab.com/esr/cvs-fast-export
cd cvs-fast-export
sudo dnf install dnf-plgins-core
sudo dnf install epel-release
sudo dnf config-manager --set-enabled powertools
sudo buildprep --doc
make install-bin
make install

[brianr@rockysmebuild cvs-fast-export]$ cvs-fast-export --help
Usage: cvs-fast-export [OPTIONS] [FILE]...
Parse RCS files and emit a fast-import stream.

Mandatory arguments to long options are mandatory for short options too.
 -h --help                       This help
 -g --graph                      Dump the commit graph
 -V --version                    Print version
 -w --commit-time-window=WINDOW  Time window for commits(seconds)
 -c --content-only               Don't trust commit-IDs
 -l --log=LOG_FILE               Log file
 -a --authorlist                 Report committer IDs from repository
 -A --authormap=AUTHOR_MAP       Author map file
 -R --revision-map=REV_MAP       Revision map file
 -r --reposurgeon                Issue cvs-revision properties
 -T                              Force deterministic dates
 -e --remote=REMOTE              Relocate branches to refs/remotes/REMOTE
 -s --strip=PREFIX               Strip the given PREFIX instead of longest common prefix
 -p --progress                   Enable load-status reporting
 -P --promiscuous                Process files without ,v extension
 -v --verbose                    Show verbose progress messages
 -q --quiet                      Suppress normal warnings
 -i --incremental=TIME           Incremental dump beginning after specified RFC3339-format TIME.
 -t --threads=N                  Use threaded scheduler with N threads for CVS master analyses.
 -E --embed-id                   Embed CVS revisions in the commit messages.

Example: find | cvs-fast-export
[brianr@rockysmebuild cvs-fast-export]$

It includes a "cvsconvert" script which runs cvs-fast-convert and writes to a git repo.

HOWEVER it needs access to the CVS files, not just the cvs client!

[brianr@rockysmebuild cvs-fast-export]$ more cvsconvert 
#!/usr/bin/env python3
"""
cvsconvert - convert a CVS repo and check against the original

Convert, and check the tree content of a gitspace conversion against
the CVS.  The tip state of every branch, and every tag, is checked.

Will produce spurious errors if any CVS branch name had to be sanitized.

cvssync (part of the cvs package) is the way to go.

See here: https://stackoverflow.com/questions/28693868/migration-from-cvs-to-git


This worked:

cvssync brianr@shell.koozali.org:/cvs/smecontribs/rpms smeserver-ddclient

it creates a directory "smeserver-ddclient" of the CVS files.

and "find . | cvs-fast-export | git fast-import" also did something, although I am not sure it was quite what I expected!!

TBC (WIP)

Retrieved from "https://wiki.koozali.org/index.php?title=First_Steps_with_Gitea&oldid=41994"